2026-03-14 08:52:16.849549 | Job console starting...
2026-03-14 08:52:16.870473 | Updating repositories
2026-03-14 08:52:16.993323 | Preparing job workspace
2026-03-14 08:52:28.118197 | Running Ansible setup...
2026-03-14 08:52:33.554866 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master]
2026-03-14 08:52:34.189911 | 
2026-03-14 08:52:34.190055 | PLAY [localhost]
2026-03-14 08:52:34.198886 | 
2026-03-14 08:52:34.198971 | TASK [Gathering Facts]
2026-03-14 08:52:35.181294 | localhost | ok
2026-03-14 08:52:35.200048 | 
2026-03-14 08:52:35.200172 | TASK [log-inventory : Ensure Zuul Ansible directory exists]
2026-03-14 08:52:35.620402 | localhost -> localhost | changed
2026-03-14 08:52:35.639637 | 
2026-03-14 08:52:35.639800 | TASK [log-inventory : Copy ansible inventory to logs dir]
2026-03-14 08:52:36.451701 | localhost -> localhost | changed
2026-03-14 08:52:36.470590 | 
2026-03-14 08:52:36.470729 | TASK [Setup log path fact]
2026-03-14 08:52:36.495009 | localhost | ok
2026-03-14 08:52:36.518858 | 
2026-03-14 08:52:36.518948 | TASK [set-zuul-log-path-fact : Set log path for a build]
2026-03-14 08:52:36.549947 | localhost | ok
2026-03-14 08:52:36.558378 | 
2026-03-14 08:52:36.558446 | TASK [emit-job-header : Print job information]
2026-03-14 08:52:36.586509 | # Job Information
2026-03-14 08:52:36.586712 | Ansible Version: 2.15.12
2026-03-14 08:52:36.586750 | Job: watcher-operator-kuttl
2026-03-14 08:52:36.586777 | Pipeline: github-check
2026-03-14 08:52:36.586802 | Executor: ze01.softwarefactory-project.io
2026-03-14 08:52:36.586824 | Triggered by: https://github.com/openstack-k8s-operators/watcher-operator/pull/353
2026-03-14 08:52:36.586846 | Log URL (when completed): https://logserver.rdoproject.org/fbf/rdoproject.org/fbfdf80f16824a47b233a8a597241091/
2026-03-14 08:52:36.586868 | Event ID: dfe96420-1f77-11f1-8e80-23c9abe81bb0
2026-03-14 08:52:36.591417 | 
2026-03-14 08:52:36.591507 | LOOP [emit-job-header : Print node information]
2026-03-14 08:52:36.708428 | localhost | ok:
2026-03-14 08:52:36.709094 | localhost | # Node Information
2026-03-14 08:52:36.709181 | localhost | Inventory Hostname: controller
2026-03-14 08:52:36.709234 | localhost | Hostname: np0005647129
2026-03-14 08:52:36.709279 | localhost | Username: zuul
2026-03-14 08:52:36.709324 | localhost | Distro: CentOS 9
2026-03-14 08:52:36.709364 | localhost | Provider: vexxhost-nodepool-tripleo
2026-03-14 08:52:36.709402 | localhost | Region: RegionOne
2026-03-14 08:52:36.709439 | localhost | Label: cloud-centos-9-stream-tripleo-vexxhost-medium
2026-03-14 08:52:36.709475 | localhost | Product Name: OpenStack Nova
2026-03-14 08:52:36.709511 | localhost | Interface IP: 38.102.83.180
2026-03-14 08:52:36.793515 | localhost | ok:
2026-03-14 08:52:36.793621 | localhost | # Node Information
2026-03-14 08:52:36.793645 | localhost | Inventory Hostname: crc
2026-03-14 08:52:36.793664 | localhost | Hostname: crc
2026-03-14 08:52:36.793681 | localhost | Username: core
2026-03-14 08:52:36.793698 | localhost | Distro: RedHat 4.18
2026-03-14 08:52:36.793714 | localhost | Provider: vexxhost-nodepool-tripleo
2026-03-14 08:52:36.793730 | localhost | Region: RegionOne
2026-03-14 08:52:36.793746 | localhost | Label: crc-cloud-ocp-4-18-1-3xl
2026-03-14 08:52:36.793762 | localhost | Product Name: OpenStack Nova
2026-03-14 08:52:36.793778 | localhost | Interface IP: 38.102.83.32
2026-03-14 08:52:36.842863 | 
2026-03-14 08:52:36.843069 | PLAY [all]
2026-03-14 08:52:36.858122 | 
2026-03-14 08:52:36.858290 | TASK [Gather network facts]
2026-03-14 08:52:37.241189 | controller | ok
2026-03-14 08:52:37.573585 | crc | ok
2026-03-14 08:52:37.624565 | 
2026-03-14 08:52:37.624708 | TASK [include_role : start-zuul-console]
2026-03-14 08:52:37.651760 | controller | ok
2026-03-14 08:52:37.667479 | crc | ok
2026-03-14 08:52:37.685141 | 
2026-03-14 08:52:37.685205 | TASK [start-zuul-console : Start zuul_console daemon.]
2026-03-14 08:52:38.045700 | controller | ok
2026-03-14 08:52:38.128145 | crc | ok
2026-03-14 08:52:38.147924 | 
2026-03-14 08:52:38.148069 | TASK [include_role : add-build-sshkey]
2026-03-14 08:52:38.184634 | controller | ok
2026-03-14 08:52:38.192046 | crc | ok
2026-03-14 08:52:38.215302 | 
2026-03-14 08:52:38.215421 | TASK [add-build-sshkey : Check to see if ssh key was already created for this build]
2026-03-14 08:52:38.495705 | controller -> localhost | ok
2026-03-14 08:52:38.541134 | 
2026-03-14 08:52:38.541275 | TASK [add-build-sshkey : Create a new key in workspace based on build UUID]
2026-03-14 08:52:38.581501 | controller | ok
2026-03-14 08:52:38.590138 | crc | ok
2026-03-14 08:52:38.616339 | controller | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/create-key-and-replace.yaml
2026-03-14 08:52:38.616463 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/create-key-and-replace.yaml
2026-03-14 08:52:38.625029 | 
2026-03-14 08:52:38.625120 | TASK [add-build-sshkey : Create Temp SSH key]
2026-03-14 08:52:39.263689 | controller -> localhost | Generating public/private rsa key pair.
2026-03-14 08:52:39.263917 | controller -> localhost | Your identification has been saved in /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/fbfdf80f16824a47b233a8a597241091_id_rsa.
2026-03-14 08:52:39.263959 | controller -> localhost | Your public key has been saved in /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/fbfdf80f16824a47b233a8a597241091_id_rsa.pub.
2026-03-14 08:52:39.263992 | controller -> localhost | The key fingerprint is:
2026-03-14 08:52:39.264045 | controller -> localhost | SHA256:cH6xhDxjm2l/pa8gvi4gNaXlFIqkB4QElMQEdFNYLwo zuul-build-sshkey
2026-03-14 08:52:39.264079 | controller -> localhost | The key's randomart image is:
2026-03-14 08:52:39.264106 | controller -> localhost | +---[RSA 3072]----+
2026-03-14 08:52:39.264133 | controller -> localhost | |/Boo+o..         |
2026-03-14 08:52:39.264160 | controller -> localhost | |.*.o..=. .       |
2026-03-14 08:52:39.264186 | controller -> localhost | |.Eo .B..B o      |
2026-03-14 08:52:39.264213 | controller -> localhost | | .. = o= B o     |
2026-03-14 08:52:39.264239 | controller -> localhost | |   o .  S o   .  |
2026-03-14 08:52:39.264266 | controller -> localhost | |  . .  . o   o   |
2026-03-14 08:52:39.264292 | controller -> localhost | |   . .  . o o    |
2026-03-14 08:52:39.264318 | controller -> localhost | |      .. . o .   |
2026-03-14 08:52:39.264344 | controller -> localhost | |       o+.  ...  |
2026-03-14 08:52:39.264369 | controller -> localhost | +----[SHA256]-----+
2026-03-14 08:52:39.264426 | controller -> localhost | ok: Runtime: 0:00:00.229500
2026-03-14 08:52:39.273656 | 
2026-03-14 08:52:39.273764 | TASK [add-build-sshkey : Remote setup ssh keys (linux)]
2026-03-14 08:52:39.310474 | controller | ok
2026-03-14 08:52:39.331279 | crc | ok
2026-03-14 08:52:39.352290 | controller | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/remote-linux.yaml
2026-03-14 08:52:39.352387 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/remote-linux.yaml
2026-03-14 08:52:39.364235 | 
2026-03-14 08:52:39.364341 | TASK [add-build-sshkey : Remove previously added zuul-build-sshkey]
2026-03-14 08:52:39.389856 | controller | skipping: Conditional result was False
2026-03-14 08:52:39.400860 | crc | skipping: Conditional result was False
2026-03-14 08:52:39.406379 | 
2026-03-14 08:52:39.406477 | TASK [add-build-sshkey : Enable access via build key on all nodes]
2026-03-14 08:52:39.803835 | controller | changed
2026-03-14 08:52:39.989767 | crc | changed
2026-03-14 08:52:40.001284 | 
2026-03-14 08:52:40.001432 | TASK [add-build-sshkey : Make sure user has a .ssh]
2026-03-14 08:52:40.268623 | controller | ok
2026-03-14 08:52:40.340365 | crc | ok
2026-03-14 08:52:40.347030 | 
2026-03-14 08:52:40.347115 | TASK [add-build-sshkey : Install build private key as SSH key on all nodes]
2026-03-14 08:52:40.981171 | controller | changed
2026-03-14 08:52:41.249398 | crc | changed
2026-03-14 08:52:41.254395 | 
2026-03-14 08:52:41.254457 | TASK [add-build-sshkey : Install build public key as SSH key on all nodes]
2026-03-14 08:52:41.897115 | controller | changed
2026-03-14 08:52:42.119291 | crc | changed
2026-03-14 08:52:42.131210 | 
2026-03-14 08:52:42.131356 | TASK [add-build-sshkey : Remote setup ssh keys (windows)]
2026-03-14 08:52:42.164108 | controller | skipping: Conditional result was False
2026-03-14 08:52:42.175006 | crc | skipping: Conditional result was False
2026-03-14 08:52:42.182751 | 
2026-03-14 08:52:42.182839 | TASK [remove-zuul-sshkey : Remove master key from local agent]
2026-03-14 08:52:42.746262 | controller -> localhost | changed
2026-03-14 08:52:42.761735 | 
2026-03-14 08:52:42.761824 | TASK [add-build-sshkey : Add back temp key]
2026-03-14 08:52:42.988095 | controller -> localhost | Identity added: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/fbfdf80f16824a47b233a8a597241091_id_rsa (zuul-build-sshkey)
2026-03-14 08:52:42.988308 | controller -> localhost | ok: Runtime: 0:00:00.007979
2026-03-14 08:52:42.994252 | 
2026-03-14 08:52:42.994313 | TASK [add-build-sshkey : Verify we can still SSH to all nodes]
2026-03-14 08:52:43.355295 | controller | ok
2026-03-14 08:52:43.386342 | crc | ok
2026-03-14 08:52:43.392459 | 
2026-03-14 08:52:43.392626 | TASK [add-build-sshkey : Verify we can still SSH to all nodes (windows)]
2026-03-14 08:52:43.439708 | crc | skipping: Conditional result was False
2026-03-14 08:52:43.440149 | controller | skipping: Conditional result was False
2026-03-14 08:52:43.454987 | 
2026-03-14 08:52:43.455071 | TASK [include_role : validate-host]
2026-03-14 08:52:43.488317 | controller | ok
2026-03-14 08:52:43.495904 | crc | ok
2026-03-14 08:52:43.524172 | 
2026-03-14 08:52:43.524264 | TASK [validate-host : Define zuul_info_dir fact]
2026-03-14 08:52:43.554753 | controller | ok
2026-03-14 08:52:43.570478 | crc | ok
2026-03-14 08:52:43.574875 | 
2026-03-14 08:52:43.574933 | TASK [validate-host : Ensure Zuul Ansible directory exists]
2026-03-14 08:52:43.811897 | controller -> localhost | ok
2026-03-14 08:52:43.817591 | 
2026-03-14 08:52:43.817657 | TASK [validate-host : Collect information about the host]
2026-03-14 08:52:44.737983 | controller | ok
2026-03-14 08:52:49.179050 | crc | ok
2026-03-14 08:52:49.192654 | 
2026-03-14 08:52:49.192822 | TASK [validate-host : Sanitize hostname]
2026-03-14 08:52:49.243676 | controller | ok
2026-03-14 08:52:49.259422 | crc | ok
2026-03-14 08:52:49.265087 | 
2026-03-14 08:52:49.265173 | TASK [validate-host : Write out all ansible variables/facts known for each host]
2026-03-14 08:52:49.722113 | controller -> localhost | changed
2026-03-14 08:52:49.805118 | crc -> localhost | changed
2026-03-14 08:52:49.812543 | 
2026-03-14 08:52:49.812635 | TASK [validate-host : Collect information about zuul worker]
2026-03-14 08:52:52.196708 | crc | ok
2026-03-14 08:52:52.993619 | controller | ok
2026-03-14 08:52:53.009360 | 
2026-03-14 08:52:53.009472 | TASK [validate-host : Write out all zuul information for each host]
2026-03-14 08:52:53.817324 | crc -> localhost | changed
2026-03-14 08:52:53.859041 | controller -> localhost | changed
2026-03-14 08:52:53.874493 | 
2026-03-14 08:52:53.874601 | TASK [include_role : prepare-workspace-openshift]
2026-03-14 08:52:53.888795 | controller | skipping: Conditional result was False
2026-03-14 08:52:53.895480 | 
2026-03-14 08:52:53.895560 | TASK [include_role : remove-zuul-sshkey]
2026-03-14 08:52:53.909850 | controller | skipping: Conditional result was False
2026-03-14 08:52:53.915566 | 
2026-03-14 08:52:53.915633 | LOOP [ensure-output-dirs : Empty Zuul Output directories by removing them]
2026-03-14 08:52:54.393757 | controller | ok: "logs"
2026-03-14 08:52:54.393984 | controller | ok: All items complete
2026-03-14 08:52:54.394053 | 
2026-03-14 08:52:54.420404 | crc | ok: "logs"
2026-03-14 08:52:54.564369 | controller | ok: "artifacts"
2026-03-14 08:52:54.698484 | crc | ok: "artifacts"
2026-03-14 08:52:54.785512 | controller | ok: "docs"
2026-03-14 08:52:54.963610 | crc | ok: "docs"
2026-03-14 08:52:54.981846 | 
2026-03-14 08:52:54.981979 | LOOP [ensure-output-dirs : Ensure Zuul Output directories exist]
2026-03-14 08:52:55.320853 | controller | changed: "logs"
2026-03-14 08:52:55.321381 | controller | ok: All items complete
2026-03-14 08:52:55.321454 | 
2026-03-14 08:52:55.335881 | crc | changed: "logs"
2026-03-14 08:52:55.492011 | controller | changed: "artifacts"
2026-03-14 08:52:55.585896 | crc | changed: "artifacts"
2026-03-14 08:52:55.668771 | controller | changed: "docs"
2026-03-14 08:52:55.831932 | crc | changed: "docs"
2026-03-14 08:52:55.939089 | 
2026-03-14 08:52:55.939186 | PLAY RECAP
2026-03-14 08:52:55.939231 | controller | ok: 22 changed: 9 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0
2026-03-14 08:52:55.939259 | crc | ok: 17 changed: 6 unreachable: 0 failed: 0 skipped: 3 rescued: 0 ignored: 0
2026-03-14 08:52:55.939283 | localhost | ok: 6 changed: 2 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0
2026-03-14 08:52:55.939301 | 
2026-03-14 08:52:56.047282 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master]
2026-03-14 08:52:56.057371 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/ci-framework-rdo-base/pre.yaml@master]
2026-03-14 08:52:56.759919 | 
2026-03-14 08:52:56.760058 | PLAY [all]
2026-03-14 08:52:56.799916 | 
2026-03-14 08:52:56.800101 | TASK [mirror-info-fork : Set mirror_fqdn if not defined]
2026-03-14 08:52:56.890202 | crc | ok
2026-03-14 08:52:56.895085 | controller | ok
2026-03-14 08:52:56.899865 | 
2026-03-14 08:52:56.899942 | TASK [mirror-info-fork : Create /etc/ci]
2026-03-14 08:52:57.289797 | controller | changed
2026-03-14 08:52:57.362135 | crc | changed
2026-03-14 08:52:57.370931 | 
2026-03-14 08:52:57.371103 | TASK [mirror-info-fork : Install ci_mirror script]
2026-03-14 08:52:58.192202 | controller | changed
2026-03-14 08:52:58.329198 | crc | changed
2026-03-14 08:52:58.368303 | 
2026-03-14 08:52:58.368492 | LOOP [add-authorized-keys : Enable access via build key on all nodes]
2026-03-14 08:52:58.776827 | controller | changed:
2026-03-14 08:52:58.777128 | controller | {
2026-03-14 08:52:58.777179 | controller |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4Z/c9osaGGtU6X8fgELwfj/yayRurfcKA0HMFfdpPxev2dbwljysMuzoVp4OZmW1gvGtyYPSNRvnzgsaabPNKNo2ym5NToCP6UM+KSe93aln4BcM/24mXChYAbXJQ5Bqq/pIzsGs/pKetQN+vwvMxLOwTvpcsCJBXaa981RKML6xj9l/UZ7IIq1HSEKMvPLxZMWdu0Ut8DkCd5F4nOw9Wgml2uYpDCj5LLCrQQ9ChdOMz8hz6SighhNlRpPkvPaet3OXxr/ytFMu7j7vv06CaEnuMMiY2aTWN1Imin9eHAylIqFHta/3gFfQSWt9jXM7owkBLKL7ATzhaAn+fjNupw== arxcruz@redhat.com"
2026-03-14 08:52:58.777214 | controller | }
2026-03-14 08:52:58.896106 | crc | changed:
2026-03-14 08:52:58.896315 | crc | {
2026-03-14 08:52:58.896343 | crc |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4Z/c9osaGGtU6X8fgELwfj/yayRurfcKA0HMFfdpPxev2dbwljysMuzoVp4OZmW1gvGtyYPSNRvnzgsaabPNKNo2ym5NToCP6UM+KSe93aln4BcM/24mXChYAbXJQ5Bqq/pIzsGs/pKetQN+vwvMxLOwTvpcsCJBXaa981RKML6xj9l/UZ7IIq1HSEKMvPLxZMWdu0Ut8DkCd5F4nOw9Wgml2uYpDCj5LLCrQQ9ChdOMz8hz6SighhNlRpPkvPaet3OXxr/ytFMu7j7vv06CaEnuMMiY2aTWN1Imin9eHAylIqFHta/3gFfQSWt9jXM7owkBLKL7ATzhaAn+fjNupw== arxcruz@redhat.com"
2026-03-14 08:52:58.896363 | crc | }
2026-03-14 08:52:59.016623 | controller | changed:
2026-03-14 08:52:59.016717 | controller | {
2026-03-14 08:52:59.016741 | controller |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDS4Fn6k4deCnIlOtLWqZJyksbepjQt04j8Ed8CGx9EKkj0fKiAxiI4TadXQYPuNHMixZy4Nevjb6aDhL5Z906TfvNHKUrjrG7G26a0k8vdc61NEQ7FmcGMWRLwwc6ReDO7lFpzYKBMk4YqfWgBuGU/K6WLKiVW2cVvwIuGIaYrE1OiiX0iVUUk7KApXlDJMXn7qjSYynfO4mF629NIp8FJal38+Kv+HA+0QkE5Y2xXnzD4Lar5+keymiCHRntPppXHeLIRzbt0gxC7v3L72hpQ3BTBEzwHpeS8KY+SX1y5lRMN45thCHfJqGmARJREDjBvWG8JXOPmVIKQtZmVcD5b mandreou@redhat.com"
2026-03-14 08:52:59.016761 | controller | }
2026-03-14 08:52:59.240050 | crc | changed:
2026-03-14 08:52:59.240162 | crc | {
2026-03-14 08:52:59.240195 | crc |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDS4Fn6k4deCnIlOtLWqZJyksbepjQt04j8Ed8CGx9EKkj0fKiAxiI4TadXQYPuNHMixZy4Nevjb6aDhL5Z906TfvNHKUrjrG7G26a0k8vdc61NEQ7FmcGMWRLwwc6ReDO7lFpzYKBMk4YqfWgBuGU/K6WLKiVW2cVvwIuGIaYrE1OiiX0iVUUk7KApXlDJMXn7qjSYynfO4mF629NIp8FJal38+Kv+HA+0QkE5Y2xXnzD4Lar5+keymiCHRntPppXHeLIRzbt0gxC7v3L72hpQ3BTBEzwHpeS8KY+SX1y5lRMN45thCHfJqGmARJREDjBvWG8JXOPmVIKQtZmVcD5b mandreou@redhat.com"
2026-03-14 08:52:59.240233 | crc | }
2026-03-14 08:52:59.253003 | controller | changed:
2026-03-14 08:52:59.253140 | controller | {
2026-03-14 08:52:59.253177 | controller |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9MiLfy30deHA7xPOAlew5qUq3UP2gmRMYJi8PtkjFB20/DKeWwWNnkZPqP9AayruRoo51SIiVg870gbZE2jYl+Ncx/FYDe56JeC3ySZsXoAVkC9bP7gkOGqOmJjirvAgPMI7bogVz8i+66Q4Ar7OKTp3762G4IuWPPEg4ce4Y7lx9qWocZapHYq4cYKMxrOZ7SEbFSATBbe2bPZAPKTw8do/Eny+Hq/LkHFhIeyra6cqTFQYShr+zPln0Cr+ro/pDX3bB+1ubFgTpjpkkkQsLhDfR6cCdCWM2lgnS3BTtYj5Ct9/JRPR5YOphqZz+uB+OEu2IL68hmU9vNTth1KeX rlandy@redhat.com"
2026-03-14 08:52:59.253205 | controller | }
2026-03-14 08:52:59.498529 | controller | changed:
2026-03-14 08:52:59.498684 | controller | {
2026-03-14 08:52:59.498739 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFCbgz8gdERiJlk2IKOtkjQxEXejrio6ZYMJAVJYpOIp raukadah@gmail.com"
2026-03-14 08:52:59.498782 | controller | }
2026-03-14 08:52:59.625884 | crc | changed:
2026-03-14 08:52:59.626055 | crc | {
2026-03-14 08:52:59.626114 | crc |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9MiLfy30deHA7xPOAlew5qUq3UP2gmRMYJi8PtkjFB20/DKeWwWNnkZPqP9AayruRoo51SIiVg870gbZE2jYl+Ncx/FYDe56JeC3ySZsXoAVkC9bP7gkOGqOmJjirvAgPMI7bogVz8i+66Q4Ar7OKTp3762G4IuWPPEg4ce4Y7lx9qWocZapHYq4cYKMxrOZ7SEbFSATBbe2bPZAPKTw8do/Eny+Hq/LkHFhIeyra6cqTFQYShr+zPln0Cr+ro/pDX3bB+1ubFgTpjpkkkQsLhDfR6cCdCWM2lgnS3BTtYj5Ct9/JRPR5YOphqZz+uB+OEu2IL68hmU9vNTth1KeX rlandy@redhat.com"
2026-03-14 08:52:59.626162 | crc | }
2026-03-14 08:52:59.740697 | controller | changed:
2026-03-14 08:52:59.740882 | controller | {
2026-03-14 08:52:59.740936 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBqb3Q/9uDf4LmihQ7xeJ9gA/STIQUFPSfyyV0m8AoQi bshewale@redhat.com"
2026-03-14 08:52:59.740978 | controller | }
2026-03-14 08:52:59.992454 | controller | changed:
2026-03-14 08:52:59.992617 | controller | {
2026-03-14 08:52:59.992663 | controller |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0I8QqQx0Az2ysJt2JuffucLijhBqnsXKEIx5GyHwxVULROa8VtNFXUDH6ZKZavhiMcmfHB2+TBTda+lDP4FldYj06dGmzCY+IYGa+uDRdxHNGYjvCfLFcmLlzRK6fNbTcui+KlUFUdKe0fb9CRoGKyhlJD5GRkM1Dv+Yb6Bj+RNnmm1fVGYxzmrD2utvffYEb0SZGWxq2R9gefx1q/3wCGjeqvufEV+AskPhVGc5T7t9eyZ4qmslkLh1/nMuaIBFcr9AUACRajsvk6mXrAN1g3HlBf2gQlhi1UEyfbqIQvzzFtsbLDlSum/KmKjy818GzvWjERfQ0VkGzCd9bSLVL dviroel@redhat.com"
2026-03-14 08:52:59.992692 | controller | }
2026-03-14 08:53:00.017198 | crc | changed:
2026-03-14 08:53:00.017295 | crc | {
2026-03-14 08:53:00.017328 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFCbgz8gdERiJlk2IKOtkjQxEXejrio6ZYMJAVJYpOIp raukadah@gmail.com"
2026-03-14 08:53:00.017354 | crc | }
2026-03-14 08:53:00.231617 | controller | changed:
2026-03-14 08:53:00.231714 | controller | {
2026-03-14 08:53:00.231738 | controller |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLOQd4ZLtkZXQGY6UwAr/06ppWQK4fDO3HaqxPk98csyOCBXsliSKK39Bso828+5srIXiW7aI6aC9P5mwi4mUZlGPfJlQbfrcGvY+b/SocuvaGK+1RrHLoJCT52LBhwgrzlXio2jeksZeein8iaTrhsPrOAs7KggIL/rB9hEiB3NaOPWhhoCP4vlW6MEMExGcqB/1FVxXFBPnLkEyW0Lk7ycVflZl2ocRxbfjZi0+tI1Wlinp8PvSQSc/WVrAcDgKjc/mB4ODPOyYy3G8FHgfMsrXSDEyjBKgLKMsdCrAUcqJQWjkqXleXSYOV4q3pzL+9umK+q/e3P/bIoSFQzmJKTU1eDfuvPXmow9F5H54fii/Da7ezlMJ+wPGHJrRAkmzvMbALy7xwswLhZMkOGNtRcPqaKYRmIBKpw3o6bCTtcNUHOtOQnzwY8JzrM2eBWJBXAANYw+9/ho80JIiwhg29CFNpVBuHbql2YxJQNrnl90guN65rYNpDxdIluweyUf8= anbanerj@kaermorhen"
2026-03-14 08:53:00.231758 | controller | }
2026-03-14 08:53:00.361450 | crc | changed:
2026-03-14 08:53:00.361558 | crc | {
2026-03-14 08:53:00.361580 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBqb3Q/9uDf4LmihQ7xeJ9gA/STIQUFPSfyyV0m8AoQi bshewale@redhat.com"
2026-03-14 08:53:00.361609 | crc | }
2026-03-14 08:53:00.458995 | controller | changed:
2026-03-14 08:53:00.459143 | controller | {
2026-03-14 08:53:00.459171 | controller |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3VwV8Im9kRm49lt3tM36hj4Zv27FxGo4C1Q/0jqhzFmHY7RHbmeRr8ObhwWoHjXSozKWg8FL5ER0z3hTwL0W6lez3sL7hUaCmSuZmG5Hnl3x4vTSxDI9JZ/Y65rtYiiWQo2fC5xJhU/4+0e5e/pseCm8cKRSu+SaxhO+sd6FDojA2x1BzOzKiQRDy/1zWGp/cZkxcEuB1wHI5LMzN03c67vmbu+fhZRAUO4dQkvcnj2LrhQtpa+ytvnSjr8icMDosf1OsbSffwZFyHB/hfWGAfe0eIeSA2XPraxiPknXxiPKx2MJsaUTYbsZcm3EjFdHBBMumw5rBI74zLrMRvCO9GwBEmGT4rFng1nP+yw5DB8sn2zqpOsPg1LYRwCPOUveC13P6pgsZZPh812e8v5EKnETct+5XI3dVpdw6CnNiLwAyVAF15DJvBGT/u1k0Myg/bQn+Gv9k2MSj6LvQmf6WbZu2Wgjm30z3FyCneBqTL7mLF19YXzeC0ufHz5pnO1E= dasm@fedora"
2026-03-14 08:53:00.459195 | controller | }
2026-03-14 08:53:00.698451 | controller | changed:
2026-03-14 08:53:00.698555 | controller | {
2026-03-14 08:53:00.698581 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHUnwjB20UKmsSed9X73eGNV5AOEFccQ3NYrRW776pEk cjeanner"
2026-03-14 08:53:00.698600 | controller | }
2026-03-14 08:53:00.756339 | crc | changed:
2026-03-14 08:53:00.756452 | crc | {
2026-03-14 08:53:00.756484 | crc |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0I8QqQx0Az2ysJt2JuffucLijhBqnsXKEIx5GyHwxVULROa8VtNFXUDH6ZKZavhiMcmfHB2+TBTda+lDP4FldYj06dGmzCY+IYGa+uDRdxHNGYjvCfLFcmLlzRK6fNbTcui+KlUFUdKe0fb9CRoGKyhlJD5GRkM1Dv+Yb6Bj+RNnmm1fVGYxzmrD2utvffYEb0SZGWxq2R9gefx1q/3wCGjeqvufEV+AskPhVGc5T7t9eyZ4qmslkLh1/nMuaIBFcr9AUACRajsvk6mXrAN1g3HlBf2gQlhi1UEyfbqIQvzzFtsbLDlSum/KmKjy818GzvWjERfQ0VkGzCd9bSLVL dviroel@redhat.com"
2026-03-14 08:53:00.756511 | crc | }
2026-03-14 08:53:01.016154 | controller | changed:
2026-03-14 08:53:01.016243 | controller | {
2026-03-14 08:53:01.016277 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDercCMGn8rW1C4P67tHgtflPdTeXlpyUJYH+6XDd2lR jgilaber@redhat.com"
2026-03-14 08:53:01.016303 | controller | }
2026-03-14 08:53:01.149428 | crc | changed:
2026-03-14 08:53:01.149675 | crc | {
2026-03-14 08:53:01.149708 | crc |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLOQd4ZLtkZXQGY6UwAr/06ppWQK4fDO3HaqxPk98csyOCBXsliSKK39Bso828+5srIXiW7aI6aC9P5mwi4mUZlGPfJlQbfrcGvY+b/SocuvaGK+1RrHLoJCT52LBhwgrzlXio2jeksZeein8iaTrhsPrOAs7KggIL/rB9hEiB3NaOPWhhoCP4vlW6MEMExGcqB/1FVxXFBPnLkEyW0Lk7ycVflZl2ocRxbfjZi0+tI1Wlinp8PvSQSc/WVrAcDgKjc/mB4ODPOyYy3G8FHgfMsrXSDEyjBKgLKMsdCrAUcqJQWjkqXleXSYOV4q3pzL+9umK+q/e3P/bIoSFQzmJKTU1eDfuvPXmow9F5H54fii/Da7ezlMJ+wPGHJrRAkmzvMbALy7xwswLhZMkOGNtRcPqaKYRmIBKpw3o6bCTtcNUHOtOQnzwY8JzrM2eBWJBXAANYw+9/ho80JIiwhg29CFNpVBuHbql2YxJQNrnl90guN65rYNpDxdIluweyUf8= anbanerj@kaermorhen"
2026-03-14 08:53:01.149754 | crc | }
2026-03-14 08:53:01.220572 | controller | changed:
2026-03-14 08:53:01.220666 | controller | {
2026-03-14 08:53:01.220690 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAMI6kkg9Wg0sG7jIJmyZemEBwUn1yzNpQQd3gnulOmZ adrianfuscoarnejo@gmail.com"
2026-03-14 08:53:01.220709 | controller | }
2026-03-14 08:53:01.434075 | controller | changed:
2026-03-14 08:53:01.434175 | controller | {
2026-03-14 08:53:01.434204 | controller |   "public_key": "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPijwpQu/3jhhhBZInXNOLEH57DrknPc3PLbsRvYyJIFzwYjX+WD4a7+nGnMYS42MuZk6TJcVqgnqofVx4isoD4= ramishra@redhat.com"
2026-03-14 08:53:01.434224 | controller | }
2026-03-14 08:53:01.495946 | crc | changed:
2026-03-14 08:53:01.496058 | crc | {
2026-03-14 08:53:01.496084 | crc |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3VwV8Im9kRm49lt3tM36hj4Zv27FxGo4C1Q/0jqhzFmHY7RHbmeRr8ObhwWoHjXSozKWg8FL5ER0z3hTwL0W6lez3sL7hUaCmSuZmG5Hnl3x4vTSxDI9JZ/Y65rtYiiWQo2fC5xJhU/4+0e5e/pseCm8cKRSu+SaxhO+sd6FDojA2x1BzOzKiQRDy/1zWGp/cZkxcEuB1wHI5LMzN03c67vmbu+fhZRAUO4dQkvcnj2LrhQtpa+ytvnSjr8icMDosf1OsbSffwZFyHB/hfWGAfe0eIeSA2XPraxiPknXxiPKx2MJsaUTYbsZcm3EjFdHBBMumw5rBI74zLrMRvCO9GwBEmGT4rFng1nP+yw5DB8sn2zqpOsPg1LYRwCPOUveC13P6pgsZZPh812e8v5EKnETct+5XI3dVpdw6CnNiLwAyVAF15DJvBGT/u1k0Myg/bQn+Gv9k2MSj6LvQmf6WbZu2Wgjm30z3FyCneBqTL7mLF19YXzeC0ufHz5pnO1E= dasm@fedora"
2026-03-14 08:53:01.496114 | crc | }
2026-03-14 08:53:01.692760 | controller | changed:
2026-03-14 08:53:01.693008 | controller | {
2026-03-14 08:53:01.693538 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICWBreHW95Wz2Toz5YwCGQwFcUG8oFYkienDh9tntmDc ralfieri@redhat.com"
2026-03-14 08:53:01.693650 | controller | }
2026-03-14 08:53:01.870589 | crc | changed:
2026-03-14 08:53:01.870694 | crc | {
2026-03-14 08:53:01.870722 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHUnwjB20UKmsSed9X73eGNV5AOEFccQ3NYrRW776pEk cjeanner"
2026-03-14 08:53:01.870747 | crc | }
2026-03-14 08:53:01.913821 | controller | changed:
2026-03-14 08:53:01.914036 | controller | {
2026-03-14 08:53:01.914081 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDK0iKdi8jQTpQrDdLVH/AAgLVYyTXF7AQ1gjc/5uT3t ykarel@yatinkarel"
2026-03-14 08:53:01.914109 | controller | }
2026-03-14 08:53:02.144347 | controller | changed:
2026-03-14 08:53:02.144453 | controller | {
2026-03-14 08:53:02.144487 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF/V/cLotA6LZeO32VL45Hd78skuA2lJA425Sm2LlQeZ fmount@horcrux"
2026-03-14 08:53:02.144514 | controller | }
2026-03-14 08:53:02.304068 | crc | changed:
2026-03-14 08:53:02.304682 | crc | {
2026-03-14 08:53:02.304752 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDercCMGn8rW1C4P67tHgtflPdTeXlpyUJYH+6XDd2lR jgilaber@redhat.com"
2026-03-14 08:53:02.304796 | crc | }
2026-03-14 08:53:02.368541 | controller | changed:
2026-03-14 08:53:02.368659 | controller | {
2026-03-14 08:53:02.368688 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDa7QCjuDMVmRPo1rREbGwzYeBCYVN+Ou/3WKXZEC6Sr"
2026-03-14 08:53:02.368712 | controller | }
2026-03-14 08:53:02.603322 | controller | changed:
2026-03-14 08:53:02.603429 | controller | {
2026-03-14 08:53:02.603465 | controller |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCfNtF7NvKl915TGsGGoseUb06Hj8L/S4toWf0hExeY+F00woL6NvBlJD0nDct+P5a22I4EhvoQCRQ8reaPCm1lybR3uiRIJsj+8zkVvLwby9LXzfZorlNG9ofjd00FEmB09uW/YvTl6Q9XwwwX6tInzIOv3TMqTHHGOL74ibbj8J/FJR0cFEyj0z4WQRvtkh32xAHl83gbuINryMt0sqRI+clj2381NKL55DRLQrVw0gsfqqxiHAnXg21qWmc4J+b9e9kiuAFQjcjwTVkwJCcg3xbPwC/qokYRby/Y5S40UUd7/jEARGXT7RZgpzTuDd1oZiCVrnrqJNPaMNdVv5MLeFdf1B7iIe5aa/fGouX7AO4SdKhZUdnJmCFAGvjC6S3JMZ2wAcUl+OHnssfmdj7XL50cLo27vjuzMtLAgSqi6N99m92WCF2s8J9aVzszX7Xz9OKZCeGsiVJp3/NdABKzSEAyM9xBD/5Vho894Sav+otpySHe3p6RUTgbB5Zu8VyZRZ/UtB3ueXxyo764yrc6qWIDqrehm84Xm9g+/jpIBzGPl07NUNJpdt/6Sgf9RIKXw/7XypO5yZfUcuFNGTxLfqjTNrtgLZNcjfav6sSdVXVcMPL//XNuRdKmVFaO76eV/oGMQGr1fGcCD+N+CpI7+Q+fCNB6VFWG4nZFuI/Iuw== averdagu@redhat.com"
2026-03-14 08:53:02.603497 | controller | }
2026-03-14 08:53:02.709683 | crc | changed:
2026-03-14 08:53:02.709785 | crc | {
2026-03-14 08:53:02.709814 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAMI6kkg9Wg0sG7jIJmyZemEBwUn1yzNpQQd3gnulOmZ adrianfuscoarnejo@gmail.com"
2026-03-14 08:53:02.709841 | crc | }
2026-03-14 08:53:02.823835 | controller | changed:
2026-03-14 08:53:02.823948 | controller | {
2026-03-14 08:53:02.823978 | controller |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq8l27xI+QlQVdS4djp9ogSoyrNE2+Ox6vKPdhSNL1J3PE5w+WCSvMz9A5gnNuH810zwbekEApbxTze/gLQJwBHA52CChfURpXrFaxY7ePXRElwKAL3mJfzBWY/c5jnNL9TCVmFJTGZkFZP3Nh+BMgZvL6xBkt3WKm6Uq18qzd9XeKcZusrA+O+uLv1fVeQnadY9RIqOCyeFYCzLWrUfTyE8x/XG0hAWIM7qpnF2cALQS2h9n4hW5ybiUN790H08wf9hFwEf5nxY9Z9dVkPFQiTSGKNBzmnCXU9skxS/xhpFjJ5duGSZdtAHe9O+nGZm9c67hxgtf8e5PDuqAdXEv2cf6e3VBAt+Bz8EKI3yosTj0oZHfwr42Yzb1l/SKy14Rggsrc9KAQlrGXan6+u2jcQqqx7l+SWmnpFiWTV9u5cWj2IgOhApOitmRBPYqk9rE2usfO0hLn/Pj/R/Nau4803e1/EikdLE7Ps95s9mX5jRDjAoUa2JwFF5RsVFyL910= ashigupt@ashigupt.remote.csb"
2026-03-14 08:53:02.824003 | controller | }
2026-03-14 08:53:03.066587 | controller | changed:
2026-03-14 08:53:03.066698 | controller | {
2026-03-14 08:53:03.066726 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOKLl0NYKwoZ/JY5KeZU8VwRAggeOxqQJeoqp3dsAaY9"
2026-03-14 08:53:03.066749 | controller | }
2026-03-14 08:53:03.132581 | crc | changed:
2026-03-14 08:53:03.132700 | crc | {
2026-03-14 08:53:03.132741 | crc |   "public_key": "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPijwpQu/3jhhhBZInXNOLEH57DrknPc3PLbsRvYyJIFzwYjX+WD4a7+nGnMYS42MuZk6TJcVqgnqofVx4isoD4= ramishra@redhat.com"
2026-03-14 08:53:03.132766 | crc | }
2026-03-14 08:53:03.281854 | controller | changed:
2026-03-14 08:53:03.281974 | controller | {
2026-03-14 08:53:03.282005 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIASASQOH2BcOyLKuuDOdWZlPi2orcjcA8q4400T73DLH evallesp@fedora"
2026-03-14 08:53:03.282052 | controller | }
2026-03-14 08:53:03.504281 | controller | changed:
2026-03-14 08:53:03.504378 | controller | {
2026-03-14 08:53:03.504401 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILeBWlamUph+jRKV2qrx1PGU7vWuGIt5+z9k96I8WehW amsinha@amsinha-mac"
2026-03-14 08:53:03.504420 | controller | }
2026-03-14 08:53:03.515368 | crc | changed:
2026-03-14 08:53:03.515462 | crc | {
2026-03-14 08:53:03.515485 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICWBreHW95Wz2Toz5YwCGQwFcUG8oFYkienDh9tntmDc ralfieri@redhat.com"
2026-03-14 08:53:03.515503 | crc | }
2026-03-14 08:53:03.731760 | controller | changed:
2026-03-14 08:53:03.731865 | controller | {
2026-03-14 08:53:03.731887 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIANvVgvJBlK3gb1yz5uef/JqIGq4HLEmY2dYA8e37swb morenod@redhat-laptop"
2026-03-14 08:53:03.731906 | controller | }
2026-03-14 08:53:03.906158 | crc | changed:
2026-03-14 08:53:03.906377 | crc | {
2026-03-14 08:53:03.906413 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDK0iKdi8jQTpQrDdLVH/AAgLVYyTXF7AQ1gjc/5uT3t ykarel@yatinkarel"
2026-03-14 08:53:03.906434 | crc | }
2026-03-14 08:53:03.970263 | controller | changed:
2026-03-14 08:53:03.970996 | controller | {
2026-03-14 08:53:03.971124 | controller |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZdI7t1cxYx65heVI24HTV4F7oQLW1zyfxHreL2TIJKxjyrUUKIFEUmTutcBlJRLNT2Eoix6x1sOw9YrchloCLcn//SGfTElr9mSc5jbjb7QXEU+zJMhtxyEJ1Po3CUGnj7ckiIXw7wcawZtrEOAQ9pH3ExYCJcEMiyNjRQZCxT3tPK+S4B95EWh5Fsrz9CkwpjNRPPH7LigCeQTM3Wc7r97utAslBUUvYceDSLA7rMgkitJE38b7rZBeYzsGQ8YYUBjTCtehqQXxCRjizbHWaaZkBU+N3zkKB6n/iCNGIO690NK7A/qb6msTijiz1PeuM8ThOsi9qXnbX5v0PoTpcFSojV7NHAQ71f0XXuS43FhZctT+Dcx44dT8Fb5vJu2cJGrk+qF8ZgJYNpRS7gPg0EG2EqjK7JMf9ULdjSu0r+KlqIAyLvtzT4eOnQipoKlb/WG5D/0ohKv7OMQ352ggfkBFIQsRXyyTCT98Ft9juqPuahi3CAQmP4H9dyE+7+Kz437PEtsxLmfm6naNmWi7Ee1DqWPwS8rEajsm4sNM4wW9gdBboJQtc0uZw0DfLj1I9r3Mc8Ol0jYtz0yNQDSzVLrGCaJlC311trU70tZ+ZkAVV6Mn8lOhSbj1cK0lvSr6ZK4dgqGl3I1eTZJJhbLNdg7UOVaiRx9543+C/p/As7w== brjackma@redhat.com"
2026-03-14 08:53:03.971174 | controller | }
2026-03-14 08:53:04.198988 | controller | changed:
2026-03-14 08:53:04.199194 | controller | {
2026-03-14 08:53:04.199307 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKwedoZ0TWPJX/z/4TAbO/kKcDZOQVgRH0hAqrL5UCI1 vcastell@redhat.com"
2026-03-14 08:53:04.199353 | controller | }
2026-03-14 08:53:04.302491 | crc | changed:
2026-03-14 08:53:04.302588 | crc | {
2026-03-14 08:53:04.302621 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF/V/cLotA6LZeO32VL45Hd78skuA2lJA425Sm2LlQeZ fmount@horcrux"
2026-03-14 08:53:04.302648 | crc | }
2026-03-14 08:53:04.424337 | controller | changed:
2026-03-14 08:53:04.424447 | controller | {
2026-03-14 08:53:04.424481 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEmv8sE8GCk6ZTPIqF0FQrttBdL3mq7rCm/IJy0xDFh7 michburk@redhat.com"
2026-03-14 08:53:04.424730 | controller | }
2026-03-14 08:53:04.659397 | controller | changed:
2026-03-14 08:53:04.659570 | controller | {
2026-03-14 08:53:04.659596 | controller |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICy6GpGEtwevXEEn4mmLR5lmSLe23dGgAvzkB9DMNbkf rsafrono@rsafrono"
2026-03-14 08:53:04.659615 | controller | }
2026-03-14 08:53:04.711239 | crc | changed:
2026-03-14 08:53:04.711437 | crc | {
2026-03-14 08:53:04.711500 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDa7QCjuDMVmRPo1rREbGwzYeBCYVN+Ou/3WKXZEC6Sr"
2026-03-14 08:53:04.711551 | crc | }
2026-03-14 08:53:04.711623 | crc | changed: All items complete
2026-03-14 08:53:04.711689 | 
2026-03-14 08:53:05.096378 | crc | changed:
2026-03-14 08:53:05.096479 | crc | {
2026-03-14 08:53:05.096503 | crc |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCfNtF7NvKl915TGsGGoseUb06Hj8L/S4toWf0hExeY+F00woL6NvBlJD0nDct+P5a22I4EhvoQCRQ8reaPCm1lybR3uiRIJsj+8zkVvLwby9LXzfZorlNG9ofjd00FEmB09uW/YvTl6Q9XwwwX6tInzIOv3TMqTHHGOL74ibbj8J/FJR0cFEyj0z4WQRvtkh32xAHl83gbuINryMt0sqRI+clj2381NKL55DRLQrVw0gsfqqxiHAnXg21qWmc4J+b9e9kiuAFQjcjwTVkwJCcg3xbPwC/qokYRby/Y5S40UUd7/jEARGXT7RZgpzTuDd1oZiCVrnrqJNPaMNdVv5MLeFdf1B7iIe5aa/fGouX7AO4SdKhZUdnJmCFAGvjC6S3JMZ2wAcUl+OHnssfmdj7XL50cLo27vjuzMtLAgSqi6N99m92WCF2s8J9aVzszX7Xz9OKZCeGsiVJp3/NdABKzSEAyM9xBD/5Vho894Sav+otpySHe3p6RUTgbB5Zu8VyZRZ/UtB3ueXxyo764yrc6qWIDqrehm84Xm9g+/jpIBzGPl07NUNJpdt/6Sgf9RIKXw/7XypO5yZfUcuFNGTxLfqjTNrtgLZNcjfav6sSdVXVcMPL//XNuRdKmVFaO76eV/oGMQGr1fGcCD+N+CpI7+Q+fCNB6VFWG4nZFuI/Iuw== averdagu@redhat.com"
2026-03-14 08:53:05.096707 | crc | }
2026-03-14 08:53:05.496777 | crc | changed:
2026-03-14 08:53:05.496890 | crc | {
2026-03-14 08:53:05.496924 | crc |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq8l27xI+QlQVdS4djp9ogSoyrNE2+Ox6vKPdhSNL1J3PE5w+WCSvMz9A5gnNuH810zwbekEApbxTze/gLQJwBHA52CChfURpXrFaxY7ePXRElwKAL3mJfzBWY/c5jnNL9TCVmFJTGZkFZP3Nh+BMgZvL6xBkt3WKm6Uq18qzd9XeKcZusrA+O+uLv1fVeQnadY9RIqOCyeFYCzLWrUfTyE8x/XG0hAWIM7qpnF2cALQS2h9n4hW5ybiUN790H08wf9hFwEf5nxY9Z9dVkPFQiTSGKNBzmnCXU9skxS/xhpFjJ5duGSZdtAHe9O+nGZm9c67hxgtf8e5PDuqAdXEv2cf6e3VBAt+Bz8EKI3yosTj0oZHfwr42Yzb1l/SKy14Rggsrc9KAQlrGXan6+u2jcQqqx7l+SWmnpFiWTV9u5cWj2IgOhApOitmRBPYqk9rE2usfO0hLn/Pj/R/Nau4803e1/EikdLE7Ps95s9mX5jRDjAoUa2JwFF5RsVFyL910= ashigupt@ashigupt.remote.csb"
2026-03-14 08:53:05.496954 | crc | }
2026-03-14 08:53:05.886560 | crc | changed:
2026-03-14 08:53:05.886849 | crc | {
2026-03-14 08:53:05.886914 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOKLl0NYKwoZ/JY5KeZU8VwRAggeOxqQJeoqp3dsAaY9"
2026-03-14 08:53:05.886957 | crc | }
2026-03-14 08:53:06.257330 | crc | changed:
2026-03-14 08:53:06.257613 | crc | {
2026-03-14 08:53:06.257701 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIASASQOH2BcOyLKuuDOdWZlPi2orcjcA8q4400T73DLH evallesp@fedora"
2026-03-14 08:53:06.257768 | crc | }
2026-03-14 08:53:06.630130 | crc | changed:
2026-03-14 08:53:06.630309 | crc | {
2026-03-14 08:53:06.630348 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILeBWlamUph+jRKV2qrx1PGU7vWuGIt5+z9k96I8WehW amsinha@amsinha-mac"
2026-03-14 08:53:06.630376 | crc | }
2026-03-14 08:53:07.028983 | crc | changed:
2026-03-14 08:53:07.029166 | crc | {
2026-03-14 08:53:07.029221 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIANvVgvJBlK3gb1yz5uef/JqIGq4HLEmY2dYA8e37swb morenod@redhat-laptop"
2026-03-14 08:53:07.029267 | crc | }
2026-03-14 08:53:07.427983 | crc | changed:
2026-03-14 08:53:07.428138 | crc | {
2026-03-14 08:53:07.428189 | crc |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZdI7t1cxYx65heVI24HTV4F7oQLW1zyfxHreL2TIJKxjyrUUKIFEUmTutcBlJRLNT2Eoix6x1sOw9YrchloCLcn//SGfTElr9mSc5jbjb7QXEU+zJMhtxyEJ1Po3CUGnj7ckiIXw7wcawZtrEOAQ9pH3ExYCJcEMiyNjRQZCxT3tPK+S4B95EWh5Fsrz9CkwpjNRPPH7LigCeQTM3Wc7r97utAslBUUvYceDSLA7rMgkitJE38b7rZBeYzsGQ8YYUBjTCtehqQXxCRjizbHWaaZkBU+N3zkKB6n/iCNGIO690NK7A/qb6msTijiz1PeuM8ThOsi9qXnbX5v0PoTpcFSojV7NHAQ71f0XXuS43FhZctT+Dcx44dT8Fb5vJu2cJGrk+qF8ZgJYNpRS7gPg0EG2EqjK7JMf9ULdjSu0r+KlqIAyLvtzT4eOnQipoKlb/WG5D/0ohKv7OMQ352ggfkBFIQsRXyyTCT98Ft9juqPuahi3CAQmP4H9dyE+7+Kz437PEtsxLmfm6naNmWi7Ee1DqWPwS8rEajsm4sNM4wW9gdBboJQtc0uZw0DfLj1I9r3Mc8Ol0jYtz0yNQDSzVLrGCaJlC311trU70tZ+ZkAVV6Mn8lOhSbj1cK0lvSr6ZK4dgqGl3I1eTZJJhbLNdg7UOVaiRx9543+C/p/As7w== brjackma@redhat.com"
2026-03-14 08:53:07.428232 | crc | }
2026-03-14 08:53:07.790164 | crc | changed:
2026-03-14 08:53:07.790307 | crc | {
2026-03-14 08:53:07.790359 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKwedoZ0TWPJX/z/4TAbO/kKcDZOQVgRH0hAqrL5UCI1 vcastell@redhat.com"
2026-03-14 08:53:07.790401 | crc | }
2026-03-14 08:53:08.193753 | crc | changed:
2026-03-14 08:53:08.193910 | crc | {
2026-03-14 08:53:08.193963 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEmv8sE8GCk6ZTPIqF0FQrttBdL3mq7rCm/IJy0xDFh7 michburk@redhat.com"
2026-03-14 08:53:08.194098 | crc | }
2026-03-14 08:53:08.545159 | crc | changed:
2026-03-14 08:53:08.545263 | crc | {
2026-03-14 08:53:08.545287 | crc |   "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICy6GpGEtwevXEEn4mmLR5lmSLe23dGgAvzkB9DMNbkf rsafrono@rsafrono"
2026-03-14 08:53:08.545306 | crc | }
2026-03-14 08:53:08.577755 | 
2026-03-14 08:53:08.577893 | TASK [Set timezone to UTC]
2026-03-14 08:53:09.112773 | controller | changed: executed `/bin/timedatectl set-timezone UTC`
2026-03-14 08:53:09.243406 | crc | ok
2026-03-14 08:53:09.255552 | 
2026-03-14 08:53:09.255700 | TASK [Create nodepool directory]
2026-03-14 08:53:09.518628 | controller | changed
2026-03-14 08:53:09.608723 | crc | changed
2026-03-14 08:53:09.622280 | 
2026-03-14 08:53:09.622428 | TASK [Create nodepool sub_nodes file]
2026-03-14 08:53:10.204298 | controller | changed
2026-03-14 08:53:10.477980 | crc | changed
2026-03-14 08:53:10.490643 | 
2026-03-14 08:53:10.490818 | TASK [Create nodepool sub_nodes_private file]
2026-03-14 08:53:11.047944 | controller | changed
2026-03-14 08:53:11.350297 | crc | changed
2026-03-14 08:53:11.362089 | 
2026-03-14 08:53:11.362243 | LOOP [Populate nodepool sub_nodes file]
2026-03-14 08:53:11.424835 | 
2026-03-14 08:53:11.425097 | LOOP [Populate nodepool sub_nodes_private file]
2026-03-14 08:53:11.476733 | 
2026-03-14 08:53:11.476891 | TASK [Create nodepool primary file]
2026-03-14 08:53:11.503042 | controller | skipping: Conditional result was False
2026-03-14 08:53:11.513856 | crc | skipping: Conditional result was False
2026-03-14 08:53:11.521080 | 
2026-03-14 08:53:11.521150 | TASK [Create nodepool node_private for this node]
2026-03-14 08:53:12.111967 | controller | changed
2026-03-14 08:53:12.406722 | crc | changed
2026-03-14 08:53:12.416079 | 
2026-03-14 08:53:12.416219 | LOOP [Copy ssh keys to nodepool directory]
2026-03-14 08:53:12.790472 | controller | ok: Item: id_rsa Runtime: 0:00:00.003440
2026-03-14 08:53:12.790900 | 
2026-03-14 08:53:12.890035 | crc | ok: Item: id_rsa Runtime: 0:00:00.011467
2026-03-14 08:53:12.941731 | controller | ok: Item: id_rsa.pub Runtime: 0:00:00.003342
2026-03-14 08:53:13.223441 | crc | ok: Item: id_rsa.pub Runtime: 0:00:00.008637
2026-03-14 08:53:13.236399 | 
2026-03-14 08:53:13.236523 | TASK [Add sudoers role for zuul-sudo-grep.sh]
2026-03-14 08:53:13.833821 | controller | changed
2026-03-14 08:53:14.218208 | crc | changed
2026-03-14 08:53:14.230557 | 
2026-03-14 08:53:14.230689 | TASK [Validate sudoers config after edits]
2026-03-14 08:53:14.545241 | controller | /etc/sudoers: parsed OK
2026-03-14 08:53:14.545337 | controller | /etc/sudoers.d/90-cloud-init-users: parsed OK
2026-03-14 08:53:14.545348 | controller | /etc/sudoers.d/zuul: parsed OK
2026-03-14 08:53:14.545541 | controller | /etc/sudoers.d/zuul-sudo-grep: parsed OK
2026-03-14 08:53:14.661572 | crc | /etc/sudoers: parsed OK
2026-03-14 08:53:14.661761 | crc | /etc/sudoers.d/coreos-sudo-group: parsed OK
2026-03-14 08:53:14.661773 | crc | /etc/sudoers.d/zuul-sudo-grep: parsed OK
2026-03-14 08:53:14.816419 | controller | ok: Runtime: 0:00:00.005651
2026-03-14 08:53:14.833008 | crc | ok: Runtime: 0:00:00.011125
2026-03-14 08:53:14.845710 | 
2026-03-14 08:53:14.845870 | TASK [Show the environment passed in to job shell scripts]
2026-03-14 08:53:15.158485 | controller | SHELL=/bin/bash
2026-03-14 08:53:15.158519 | controller | ZUUL_PROJECT=openstack-k8s-operators/watcher-operator
2026-03-14 08:53:15.158526 | controller | ZUUL_SHORT_PROJECT_NAME=watcher-operator
2026-03-14 08:53:15.158533 | controller | ZUUL_CHANGES=openstack-k8s-operators/watcher-operator:18.0-fr5:refs/changes/53/353/3fad4a9eb56718f26ce2ec186bb570f2695f01c3
2026-03-14 08:53:15.158539 | controller | PWD=/home/zuul
2026-03-14 08:53:15.158546 | controller | ZUUL_PIPELINE=github-check
2026-03-14 08:53:15.158552 | controller | LOGNAME=zuul
2026-03-14 08:53:15.158558 | controller | XDG_SESSION_TYPE=tty
2026-03-14 08:53:15.158564 | controller | _=/usr/bin/env
2026-03-14 08:53:15.158571 | controller | MOTD_SHOWN=pam
2026-03-14 08:53:15.158576 | controller | HOME=/home/zuul
2026-03-14 08:53:15.158580 | controller | LANG=en_US.UTF-8
2026-03-14 08:53:15.158585 | controller | SSH_CONNECTION=38.102.83.114 48366 38.102.83.180 22
2026-03-14 08:53:15.158589 | controller | TOX_TESTENV_PASSENV=ZUUL_UUID ZUUL_PROJECT ZUUL_SHORT_PROJECT_NAME ZUUL_PIPELINE ZUUL_VOTING WORKSPACE BUILD_TIMEOUT ZUUL_BRANCH ZUUL_CHANGES ZUUL_CHANGE_IDS ZUUL_CHANGE ZUUL_PATCHSET
2026-03-14 08:53:15.158596 | controller | ZUUL_CHANGE_IDS=353,3fad4a9eb56718f26ce2ec186bb570f2695f01c3
2026-03-14 08:53:15.158600 | controller | WORKSPACE=/home/zuul/workspace
2026-03-14 08:53:15.158605 | controller | XDG_SESSION_CLASS=user
2026-03-14 08:53:15.158610 | controller | SELINUX_ROLE_REQUESTED=
2026-03-14 08:53:15.158614 | controller | LESSOPEN=||/usr/bin/lesspipe.sh %s
2026-03-14 08:53:15.158620 | controller | USER=zuul
2026-03-14 08:53:15.158624 | controller | ZUUL_VOTING=True
2026-03-14 08:53:15.158629 | controller | BUILD_TIMEOUT=7200000
2026-03-14 08:53:15.158633 | controller | SELINUX_USE_CURRENT_RANGE=
2026-03-14 08:53:15.158637 | controller | SHLVL=1
2026-03-14 08:53:15.158642 | controller | ZUUL_PATCHSET=3fad4a9eb56718f26ce2ec186bb570f2695f01c3
2026-03-14 08:53:15.158647 | controller | XDG_SESSION_ID=1
2026-03-14 08:53:15.158652 | controller | ZUUL_BRANCH=18.0-fr5
2026-03-14 08:53:15.158656 | controller | XDG_RUNTIME_DIR=/run/user/1000
2026-03-14 08:53:15.158661 | controller | SSH_CLIENT=38.102.83.114 48366 22
2026-03-14 08:53:15.158665 | controller | DEBUGINFOD_URLS=https://debuginfod.centos.org/
2026-03-14 08:53:15.158670 | controller | DEBUGINFOD_IMA_CERT_PATH=/etc/keys/ima:
2026-03-14 08:53:15.158674 | controller | which_declare=declare -f
2026-03-14 08:53:15.158679 | controller | PATH=/home/zuul/.local/bin:/home/zuul/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
2026-03-14 08:53:15.158683 | controller | SELINUX_LEVEL_REQUESTED=
2026-03-14 08:53:15.158688 | controller | ZUUL_CHANGE=353
2026-03-14 08:53:15.158692 | controller | DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
2026-03-14 08:53:15.158697 | controller | ZUUL_UUID=fbfdf80f16824a47b233a8a597241091
2026-03-14 08:53:15.158701 | controller | BASH_FUNC_which%%=() {  ( alias;
2026-03-14 08:53:15.158705 | controller |  eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@
2026-03-14 08:53:15.158710 | controller | }
2026-03-14 08:53:15.233393 | crc | SHELL=/bin/bash
2026-03-14 08:53:15.233479 | crc | ZUUL_PROJECT=openstack-k8s-operators/watcher-operator
2026-03-14 08:53:15.233498 | crc | ZUUL_SHORT_PROJECT_NAME=watcher-operator
2026-03-14 08:53:15.233540 | crc | ZUUL_CHANGES=openstack-k8s-operators/watcher-operator:18.0-fr5:refs/changes/53/353/3fad4a9eb56718f26ce2ec186bb570f2695f01c3
2026-03-14 08:53:15.233557 | crc | PWD=/var/home/core
2026-03-14 08:53:15.233582 | crc | ZUUL_PIPELINE=github-check
2026-03-14 08:53:15.233670 | crc | LOGNAME=core
2026-03-14 08:53:15.233686 | crc | XDG_SESSION_TYPE=tty
2026-03-14 08:53:15.233699 | crc | _=/usr/bin/env
2026-03-14 08:53:15.233712 | crc | MOTD_SHOWN=pam
2026-03-14 08:53:15.233724 | crc | HOME=/var/home/core
2026-03-14 08:53:15.233736 | crc | LANG=C.UTF-8
2026-03-14 08:53:15.233749 | crc | SSH_CONNECTION=38.102.83.114 49528 38.102.83.32 22
2026-03-14 08:53:15.233762 | crc | TOX_TESTENV_PASSENV=ZUUL_UUID ZUUL_PROJECT ZUUL_SHORT_PROJECT_NAME ZUUL_PIPELINE ZUUL_VOTING WORKSPACE BUILD_TIMEOUT ZUUL_BRANCH ZUUL_CHANGES ZUUL_CHANGE_IDS ZUUL_CHANGE ZUUL_PATCHSET
2026-03-14 08:53:15.233781 | crc | ZUUL_CHANGE_IDS=353,3fad4a9eb56718f26ce2ec186bb570f2695f01c3
2026-03-14 08:53:15.233793 | crc | WORKSPACE=/home/zuul/workspace
2026-03-14 08:53:15.233806 | crc | XDG_SESSION_CLASS=user
2026-03-14 08:53:15.233821 | crc | SELINUX_ROLE_REQUESTED=
2026-03-14 08:53:15.233834 | crc | LESSOPEN=||/usr/bin/lesspipe.sh %s
2026-03-14 08:53:15.233846 | crc | USER=core
2026-03-14 08:53:15.233859 | crc | ZUUL_VOTING=True
2026-03-14 08:53:15.233871 | crc | BUILD_TIMEOUT=7200000
2026-03-14 08:53:15.233884 | crc | SELINUX_USE_CURRENT_RANGE=
2026-03-14 08:53:15.233896 | crc | SHLVL=1
2026-03-14 08:53:15.233909 | crc | ZUUL_PATCHSET=3fad4a9eb56718f26ce2ec186bb570f2695f01c3
2026-03-14 08:53:15.233921 | crc | XDG_SESSION_ID=2
2026-03-14 08:53:15.233934 | crc | ZUUL_BRANCH=18.0-fr5
2026-03-14 08:53:15.233946 | crc | XDG_RUNTIME_DIR=/run/user/1000
2026-03-14 08:53:15.233959 | crc | SSH_CLIENT=38.102.83.114 49528 22
2026-03-14 08:53:15.233971 | crc | which_declare=declare -f
2026-03-14 08:53:15.233984 | crc | PATH=/var/home/core/.local/bin:/var/home/core/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
2026-03-14 08:53:15.233997 | crc | SELINUX_LEVEL_REQUESTED=
2026-03-14 08:53:15.234010 | crc | ZUUL_CHANGE=353
2026-03-14 08:53:15.234023 | crc | DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
2026-03-14 08:53:15.234035 | crc | ZUUL_UUID=fbfdf80f16824a47b233a8a597241091
2026-03-14 08:53:15.234051 | crc | BASH_FUNC_which%%=() {  ( alias;
2026-03-14 08:53:15.234064 | crc |  eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@
2026-03-14 08:53:15.234077 | crc | }
2026-03-14 08:53:15.449904 | controller | ok: Runtime: 0:00:00.006355
2026-03-14 08:53:15.464980 | crc | ok: Runtime: 0:00:00.012726
2026-03-14 08:53:15.476784 | 
2026-03-14 08:53:15.476926 | TASK [Workaround hardcoded /home/zuul/workspace in zuul_legacy_vars]
2026-03-14 08:53:15.507815 | controller | skipping: Conditional result was False
2026-03-14 08:53:15.529317 | crc | skipping: Conditional result was False
2026-03-14 08:53:15.538049 | 
2026-03-14 08:53:15.538147 | TASK [Symlink /home/zuul-worker/workspace]
2026-03-14 08:53:16.123276 | controller | skipping: Conditional result was False
2026-03-14 08:53:16.123719 | crc | skipping: Conditional result was False
2026-03-14 08:53:16.129249 | 
2026-03-14 08:53:16.129312 | TASK [Ensure legacy workspace directory]
2026-03-14 08:53:16.426854 | controller | changed
2026-03-14 08:53:16.452651 | crc | changed
2026-03-14 08:53:16.586933 | 
2026-03-14 08:53:16.587115 | PLAY RECAP
2026-03-14 08:53:16.587182 | controller | ok: 14 changed: 11 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0
2026-03-14 08:53:16.587236 | crc | ok: 14 changed: 10 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0
2026-03-14 08:53:16.587267 | 
2026-03-14 08:53:16.722125 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/ci-framework-rdo-base/pre.yaml@master]
2026-03-14 08:53:16.731950 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network.yml@master]
2026-03-14 08:53:17.395462 | 
2026-03-14 08:53:17.395631 | PLAY [controller]
2026-03-14 08:53:17.421232 | 
2026-03-14 08:53:17.421367 | TASK [Debug parameters]
2026-03-14 08:53:17.460661 | controller | ok
2026-03-14 08:53:17.466517 | 
2026-03-14 08:53:17.466645 | TASK [Assert crc_ci_bootstrap_cloud_name is defined]
2026-03-14 08:53:17.516315 | controller | ok: All assertions passed
2026-03-14 08:53:17.522076 | 
2026-03-14 08:53:17.522244 | TASK [Assert crc_ci_bootstrap_networking is defined and contains the minimum fields]
2026-03-14 08:53:17.592219 | controller | ok: All assertions passed
2026-03-14 08:53:17.597177 | 
2026-03-14 08:53:17.597257 | TASK [Assert that the default network exists and has no VLAN tag]
2026-03-14 08:53:17.638077 | controller | ok: All assertions passed
2026-03-14 08:53:17.643282 | 
2026-03-14 08:53:17.643353 | LOOP [Assert that all instances has a given fixed IP for each net and default net is present]
2026-03-14 08:53:17.763841 | controller | ok: All assertions passed
2026-03-14 08:53:17.771927 | controller | ok: All assertions passed
2026-03-14 08:53:17.787680 | 
2026-03-14 08:53:17.787828 | LOOP [Assert that every host references an valid/declared network]
2026-03-14 08:53:17.848801 | controller | ok: All assertions passed
2026-03-14 08:53:17.849071 | controller | ok: All items complete
2026-03-14 08:53:17.849102 | 
2026-03-14 08:53:17.854738 | controller | ok: All assertions passed
2026-03-14 08:53:17.870323 | 
2026-03-14 08:53:17.870488 | LOOP [Assert that all instances has in range fixed IP for each net]
2026-03-14 08:53:17.956678 | controller | ok: All assertions passed
2026-03-14 08:53:17.974121 | controller | ok: All assertions passed
2026-03-14 08:53:17.992557 | 
2026-03-14 08:53:17.992756 | TASK [Assert that all non-default nets has the vlan tag]
2026-03-14 08:53:18.075796 | controller | ok: All assertions passed
2026-03-14 08:53:18.097326 | 
2026-03-14 08:53:18.097559 | TASK [Install required packages]
2026-03-14 08:53:33.817723 | controller | changed
2026-03-14 08:53:33.831702 | 
2026-03-14 08:53:33.831865 | TASK [Create openstack config dir]
2026-03-14 08:53:34.192177 | controller | changed
2026-03-14 08:53:34.197575 | 
2026-03-14 08:53:34.197648 | LOOP [Create ci dir in etc folder in the controller]
2026-03-14 08:53:39.478380 | controller -> 38.102.83.32 | changed: "crc"
2026-03-14 08:53:39.479112 | controller -> 38.102.83.32 | ok: All items complete
2026-03-14 08:53:39.479277 | 
2026-03-14 08:53:41.747867 | controller -> 38.102.83.180 | changed: "controller"
2026-03-14 08:53:41.762407 | 
2026-03-14 08:53:41.762482 | TASK [Set global resource names as facts]
2026-03-14 08:53:41.795351 | controller | ok
2026-03-14 08:53:41.802124 | 
2026-03-14 08:53:41.802202 | TASK [Generate CA cert file]
2026-03-14 08:53:41.828201 | controller | skipping: Conditional result was False
2026-03-14 08:53:41.835402 | 
2026-03-14 08:53:41.835471 | TASK [Update CA certs]
2026-03-14 08:53:42.056376 | controller | skipping: Conditional result was False
2026-03-14 08:53:42.062874 | 
2026-03-14 08:53:42.062946 | TASK [Generate clouds config from cloud_secrets secret]
2026-03-14 08:53:44.951314 | controller | changed
2026-03-14 08:53:44.960939 | 
2026-03-14 08:53:44.961203 | TASK [Create the default network]
2026-03-14 08:53:47.716026 | controller | admin_state_up: true
2026-03-14 08:53:47.716110 | controller | availability_zone_hints:
2026-03-14 08:53:47.716218 | controller | - nova
2026-03-14 08:53:47.716227 | controller | availability_zones: []
2026-03-14 08:53:47.716423 | controller | created_at: '2026-03-14T08:53:46Z'
2026-03-14 08:53:47.716516 | controller | description: ''
2026-03-14 08:53:47.716608 | controller | dns_domain: ''
2026-03-14 08:53:47.716753 | controller | id: 9dc2d874-9779-4639-a8a6-76b84f2f8f52
2026-03-14 08:53:47.716885 | controller | ipv4_address_scope: null
2026-03-14 08:53:47.716988 | controller | ipv6_address_scope: null
2026-03-14 08:53:47.717175 | controller | is_default: false
2026-03-14 08:53:47.717274 | controller | is_vlan_qinq: null
2026-03-14 08:53:47.717381 | controller | is_vlan_transparent: true
2026-03-14 08:53:47.717514 | controller | l2_adjacency: true
2026-03-14 08:53:47.717633 | controller | mtu: 1500
2026-03-14 08:53:47.717733 | controller | name: zuul-ci-net-fbfdf80f
2026-03-14 08:53:47.717837 | controller | port_security_enabled: false
2026-03-14 08:53:47.717951 | controller | project_id: 4b633c451ac74233be3721a3635275e5
2026-03-14 08:53:47.718048 | controller | provider:network_type: null
2026-03-14 08:53:47.718151 | controller | provider:physical_network: null
2026-03-14 08:53:47.718250 | controller | provider:segmentation_id: null
2026-03-14 08:53:47.718359 | controller | qos_policy_id: null
2026-03-14 08:53:47.718455 | controller | revision_number: 1
2026-03-14 08:53:47.718552 | controller | router:external: false
2026-03-14 08:53:47.718643 | controller | segments: null
2026-03-14 08:53:47.718731 | controller | shared: false
2026-03-14 08:53:47.718819 | controller | status: ACTIVE
2026-03-14 08:53:47.718910 | controller | subnets: []
2026-03-14 08:53:47.718996 | controller | tags: []
2026-03-14 08:53:47.719107 | controller | updated_at: '2026-03-14T08:53:47Z'
2026-03-14 08:53:48.011275 | controller | changed
2026-03-14 08:53:48.019974 | 
2026-03-14 08:53:48.020115 | TASK [Create subnet - ipv4]
2026-03-14 08:53:50.127149 | controller | allocation_pools:
2026-03-14 08:53:50.127506 | controller | - end: 192.168.122.254
2026-03-14 08:53:50.127525 | controller |   start: 192.168.122.2
2026-03-14 08:53:50.127536 | controller | cidr: 192.168.122.0/24
2026-03-14 08:53:50.127726 | controller | created_at: '2026-03-14T08:53:49Z'
2026-03-14 08:53:50.127889 | controller | description: ''
2026-03-14 08:53:50.128056 | controller | dns_nameservers: []
2026-03-14 08:53:50.128272 | controller | dns_publish_fixed_ip: null
2026-03-14 08:53:50.128497 | controller | enable_dhcp: false
2026-03-14 08:53:50.128677 | controller | gateway_ip: 192.168.122.1
2026-03-14 08:53:50.128835 | controller | host_routes: []
2026-03-14 08:53:50.129011 | controller | id: a281b98f-17e4-4b27-b77d-2827e0a63895
2026-03-14 08:53:50.129201 | controller | ip_version: 4
2026-03-14 08:53:50.129381 | controller | ipv6_address_mode: null
2026-03-14 08:53:50.129552 | controller | ipv6_ra_mode: null
2026-03-14 08:53:50.129728 | controller | name: zuul-ci-subnet-fbfdf80f
2026-03-14 08:53:50.129918 | controller | network_id: 9dc2d874-9779-4639-a8a6-76b84f2f8f52
2026-03-14 08:53:50.130100 | controller | project_id: 4b633c451ac74233be3721a3635275e5
2026-03-14 08:53:50.130269 | controller | revision_number: 0
2026-03-14 08:53:50.130447 | controller | segment_id: null
2026-03-14 08:53:50.130628 | controller | service_types: []
2026-03-14 08:53:50.130788 | controller | subnetpool_id: null
2026-03-14 08:53:50.130928 | controller | tags: []
2026-03-14 08:53:50.131100 | controller | updated_at: '2026-03-14T08:53:49Z'
2026-03-14 08:53:50.563297 | controller | changed
2026-03-14 08:53:50.577161 | 
2026-03-14 08:53:50.577403 | TASK [Set yaml returned data for further usage]
2026-03-14 08:53:50.655370 | controller | ok
2026-03-14 08:53:50.663327 | 
2026-03-14 08:53:50.663413 | TASK [Create router]
2026-03-14 08:53:51.204698 | controller | skipping: Conditional result was False
2026-03-14 08:53:51.213597 | 
2026-03-14 08:53:51.213697 | TASK [Set router yaml returned data for further usage]
2026-03-14 08:53:51.259741 | controller | skipping: Conditional result was False
2026-03-14 08:53:51.273485 | 
2026-03-14 08:53:51.273663 | TASK [Add router to subnet]
2026-03-14 08:53:51.807930 | controller | skipping: Conditional result was False
2026-03-14 08:53:51.861486 | 
2026-03-14 08:53:51.861627 | LOOP [Create ports, VLANs and configure them on the instance]
2026-03-14 08:53:51.935829 | controller | ok:
2026-03-14 08:53:51.936118 | controller | {
2026-03-14 08:53:51.936153 | controller |   "key": "controller",
2026-03-14 08:53:51.936173 | controller |   "value": {
2026-03-14 08:53:51.936191 | controller |     "networks": {
2026-03-14 08:53:51.936208 | controller |       "default": {
2026-03-14 08:53:51.936225 | controller |         "ip": "192.168.122.11"
2026-03-14 08:53:51.936240 | controller |       }
2026-03-14 08:53:51.936257 | controller |     }
2026-03-14 08:53:51.936273 | controller |   }
2026-03-14 08:53:51.936288 | controller | }
2026-03-14 08:53:54.946109 | controller | ok:
2026-03-14 08:53:54.946279 | controller | {
2026-03-14 08:53:54.946345 | controller |   "key": "crc",
2026-03-14 08:53:54.946398 | controller |   "value": {
2026-03-14 08:53:54.946447 | controller |     "networks": {
2026-03-14 08:53:54.946494 | controller |       "default": {
2026-03-14 08:53:54.946545 | controller |         "ip": "192.168.122.10"
2026-03-14 08:53:54.946595 | controller |       },
2026-03-14 08:53:54.946641 | controller |       "internal-api": {
2026-03-14 08:53:54.946687 | controller |         "ip": "172.17.0.5"
2026-03-14 08:53:54.946733 | controller |       },
2026-03-14 08:53:54.946778 | controller |       "storage": {
2026-03-14 08:53:54.946823 | controller |         "ip": "172.18.0.5"
2026-03-14 08:53:54.946868 | controller |       },
2026-03-14 08:53:54.946912 | controller |       "tenant": {
2026-03-14 08:53:54.946956 | controller |         "ip": "172.19.0.5"
2026-03-14 08:53:54.947002 | controller |       }
2026-03-14 08:53:54.947112 | controller |     }
2026-03-14 08:53:54.947170 | controller |   }
2026-03-14 08:53:54.947218 | controller | }
2026-03-14 08:53:54.967273 | controller | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-instance.yml
2026-03-14 08:53:54.972783 | controller | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-instance.yml
2026-03-14 08:53:54.988992 | 
2026-03-14 08:53:54.989202 | TASK [Create instance ports and attach if necessary]
2026-03-14 08:53:55.022875 | controller | ok
2026-03-14 08:53:55.079966 | controller | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-instance-create-attach-port.yml
2026-03-14 08:53:55.085898 | 
2026-03-14 08:53:55.085961 | TASK [Print the host networking data]
2026-03-14 08:53:55.124894 | controller | ok
2026-03-14 08:53:55.129873 | 
2026-03-14 08:53:55.129937 | TASK [Create the private network parent port]
2026-03-14 08:53:57.673966 | controller | admin_state_up: true
2026-03-14 08:53:57.674170 | controller | allowed_address_pairs: []
2026-03-14 08:53:57.674422 | controller | binding_host_id: null
2026-03-14 08:53:57.674588 | controller | binding_profile: {}
2026-03-14 08:53:57.674751 | controller | binding_vif_details: {}
2026-03-14 08:53:57.674915 | controller | binding_vif_type: null
2026-03-14 08:53:57.675126 | controller | binding_vnic_type: normal
2026-03-14 08:53:57.675310 | controller | created_at: '2026-03-14T08:53:56Z'
2026-03-14 08:53:57.675524 | controller | data_plane_status: null
2026-03-14 08:53:57.675679 | controller | description: ''
2026-03-14 08:53:57.675830 | controller | device_id: ''
2026-03-14 08:53:57.675986 | controller | device_owner: ''
2026-03-14 08:53:57.676151 | controller | device_profile: null
2026-03-14 08:53:57.676604 | controller | dns_assignment:
2026-03-14 08:53:57.676760 | controller | - fqdn: host-192-168-122-11.openstacklocal.
2026-03-14 08:53:57.676772 | controller |   hostname: host-192-168-122-11
2026-03-14 08:53:57.676782 | controller |   ip_address: 192.168.122.11
2026-03-14 08:53:57.676790 | controller | dns_domain: ''
2026-03-14 08:53:57.676935 | controller | dns_name: ''
2026-03-14 08:53:57.677089 | controller | extra_dhcp_opts: []
2026-03-14 08:53:57.677458 | controller | fixed_ips:
2026-03-14 08:53:57.677652 | controller | - ip_address: 192.168.122.11
2026-03-14 08:53:57.677665 | controller |   subnet_id: a281b98f-17e4-4b27-b77d-2827e0a63895
2026-03-14 08:53:57.677674 | controller | hardware_offload_type: null
2026-03-14 08:53:57.677814 | controller | hints: ''
2026-03-14 08:53:57.677980 | controller | id: 3fb3d0b3-3be6-4ea1-b59d-3121f46e758b
2026-03-14 08:53:57.678113 | controller | ip_allocation: immediate
2026-03-14 08:53:57.678257 | controller | mac_address: fa:16:3e:ac:8d:09
2026-03-14 08:53:57.678472 | controller | name: controller-a1239aa3-61dc-4384-a93b-9c354d165e80
2026-03-14 08:53:57.678666 | controller | network_id: 9dc2d874-9779-4639-a8a6-76b84f2f8f52
2026-03-14 08:53:57.678830 | controller | numa_affinity_policy: null
2026-03-14 08:53:57.679040 | controller | port_security_enabled: false
2026-03-14 08:53:57.679210 | controller | project_id: 4b633c451ac74233be3721a3635275e5
2026-03-14 08:53:57.679397 | controller | propagate_uplink_status: null
2026-03-14 08:53:57.679558 | controller | resource_request: null
2026-03-14 08:53:57.679746 | controller | revision_number: 1
2026-03-14 08:53:57.679893 | controller | qos_network_policy_id: null
2026-03-14 08:53:57.680065 | controller | qos_policy_id: null
2026-03-14 08:53:57.680191 | controller | security_group_ids: []
2026-03-14 08:53:57.680345 | controller | status: DOWN
2026-03-14 08:53:57.680496 | controller | tags: []
2026-03-14 08:53:57.680648 | controller | trunk_details: null
2026-03-14 08:53:57.680799 | controller | trusted: null
2026-03-14 08:53:57.680975 | controller | updated_at: '2026-03-14T08:53:57Z'
2026-03-14 08:53:58.501730 | controller | changed
2026-03-14 08:53:58.508798 | 
2026-03-14 08:53:58.508877 | TASK [Set common facts for further usage]
2026-03-14 08:53:58.571786 | controller | ok
2026-03-14 08:53:58.578206 | 
2026-03-14 08:53:58.578278 | TASK [Create the trunk port]
2026-03-14 08:53:59.234648 | controller | skipping: Conditional result was False
2026-03-14 08:53:59.242719 | 
2026-03-14 08:53:59.242806 | TASK [Avoid NetworkManager to automatically picked up the new interface]
2026-03-14 08:53:59.299156 | controller | skipping: Conditional result was False
2026-03-14 08:53:59.307970 | 
2026-03-14 08:53:59.308151 | TASK [Refresh NetworkManager]
2026-03-14 08:53:59.356419 | controller | skipping: Conditional result was False
2026-03-14 08:53:59.364368 | 
2026-03-14 08:53:59.364497 | TASK [Attach the port to the instance]
2026-03-14 08:54:07.361803 | controller | changed
2026-03-14 08:54:07.369432 | 
2026-03-14 08:54:07.369516 | TASK [Wait till the interface exists]
2026-03-14 08:54:11.272740 | controller -> 38.102.83.180 | changed
2026-03-14 08:54:11.279875 | 
2026-03-14 08:54:11.279941 | TASK [Prepare interface data to be used configuring trunk and vlan ifaces]
2026-03-14 08:54:11.393743 | controller | ok
2026-03-14 08:54:11.399004 | 
2026-03-14 08:54:11.399079 | LOOP [Create VLANs ports and attach them to the trunk]
2026-03-14 08:54:11.418381 | 
2026-03-14 08:54:11.418500 | LOOP [Prepare VLANs interface data to be used configuring]
2026-03-14 08:54:11.438011 | 
2026-03-14 08:54:11.438144 | LOOP [Create NetworkManager configuration file for the trunk port]
2026-03-14 08:54:14.019790 | controller -> 38.102.83.180 | changed:
2026-03-14 08:54:14.020007 | controller -> 38.102.83.180 | {
2026-03-14 08:54:14.020057 | controller -> 38.102.83.180 |   "key": "default",
2026-03-14 08:54:14.020078 | controller -> 38.102.83.180 |   "value": {
2026-03-14 08:54:14.020096 | controller -> 38.102.83.180 |     "connection": "ci-private-network",
2026-03-14 08:54:14.020114 | controller -> 38.102.83.180 |     "gw": "192.168.122.1",
2026-03-14 08:54:14.020131 | controller -> 38.102.83.180 |     "iface": "eth1",
2026-03-14 08:54:14.020147 | controller -> 38.102.83.180 |     "ip": "192.168.122.11/24",
2026-03-14 08:54:14.020163 | controller -> 38.102.83.180 |     "mac": "fa:16:3e:ac:8d:09",
2026-03-14 08:54:14.020180 | controller -> 38.102.83.180 |     "mtu": "1500"
2026-03-14 08:54:14.020196 | controller -> 38.102.83.180 |   }
2026-03-14 08:54:14.020213 | controller -> 38.102.83.180 | }
2026-03-14 08:54:14.020242 | 
2026-03-14 08:54:14.032880 | 
2026-03-14 08:54:14.032995 | TASK [Refresh NetworkManager]
2026-03-14 08:54:15.221067 | controller -> 38.102.83.180 | changed
2026-03-14 08:54:15.234432 | 
2026-03-14 08:54:15.234553 | TASK [Debug fetch IP routes]
2026-03-14 08:54:15.496597 | controller -> 38.102.83.180 | changed
2026-03-14 08:54:15.511895 | 
2026-03-14 08:54:15.512141 | TASK [Debug IP routes]
2026-03-14 08:54:15.562471 | controller -> 38.102.83.180 | ok
2026-03-14 08:54:15.576086 | 
2026-03-14 08:54:15.576221 | TASK [Create instance ports and attach if necessary]
2026-03-14 08:54:15.603301 | controller | ok
2026-03-14 08:54:15.625902 | controller | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-instance-create-attach-port.yml
2026-03-14 08:54:15.636258 | 
2026-03-14 08:54:15.636432 | TASK [Print the host networking data]
2026-03-14 08:54:15.674428 | controller | ok
2026-03-14 08:54:15.683597 | 
2026-03-14 08:54:15.683733 | TASK [Create the private network parent port]
2026-03-14 08:54:18.122983 | controller | admin_state_up: true
2026-03-14 08:54:18.123098 | controller | allowed_address_pairs: []
2026-03-14 08:54:18.123354 | controller | binding_host_id: null
2026-03-14 08:54:18.123498 | controller | binding_profile: {}
2026-03-14 08:54:18.123644 | controller | binding_vif_details: {}
2026-03-14 08:54:18.123791 | controller | binding_vif_type: null
2026-03-14 08:54:18.123986 | controller | binding_vnic_type: normal
2026-03-14 08:54:18.124153 | controller | created_at: '2026-03-14T08:54:17Z'
2026-03-14 08:54:18.124330 | controller | data_plane_status: null
2026-03-14 08:54:18.124468 | controller | description: ''
2026-03-14 08:54:18.124600 | controller | device_id: ''
2026-03-14 08:54:18.124734 | controller | device_owner: ''
2026-03-14 08:54:18.124897 | controller | device_profile: null
2026-03-14 08:54:18.125424 | controller | dns_assignment:
2026-03-14 08:54:18.125625 | controller | - fqdn: host-192-168-122-10.openstacklocal.
2026-03-14 08:54:18.125641 | controller |   hostname: host-192-168-122-10
2026-03-14 08:54:18.125657 | controller |   ip_address: 192.168.122.10
2026-03-14 08:54:18.125671 | controller | dns_domain: ''
2026-03-14 08:54:18.125814 | controller | dns_name: ''
2026-03-14 08:54:18.126006 | controller | extra_dhcp_opts: []
2026-03-14 08:54:18.126404 | controller | fixed_ips:
2026-03-14 08:54:18.126565 | controller | - ip_address: 192.168.122.10
2026-03-14 08:54:18.126579 | controller |   subnet_id: a281b98f-17e4-4b27-b77d-2827e0a63895
2026-03-14 08:54:18.126593 | controller | hardware_offload_type: null
2026-03-14 08:54:18.126710 | controller | hints: ''
2026-03-14 08:54:18.126909 | controller | id: 022709b1-dd59-437e-b00d-049cb0e9bf02
2026-03-14 08:54:18.127057 | controller | ip_allocation: immediate
2026-03-14 08:54:18.127240 | controller | mac_address: fa:16:3e:e3:c1:c5
2026-03-14 08:54:18.127429 | controller | name: crc-6518bcde-aa50-4603-92c7-71dcf31294f9
2026-03-14 08:54:18.127602 | controller | network_id: 9dc2d874-9779-4639-a8a6-76b84f2f8f52
2026-03-14 08:54:18.127748 | controller | numa_affinity_policy: null
2026-03-14 08:54:18.127945 | controller | port_security_enabled: false
2026-03-14 08:54:18.128132 | controller | project_id: 4b633c451ac74233be3721a3635275e5
2026-03-14 08:54:18.128283 | controller | propagate_uplink_status: null
2026-03-14 08:54:18.128450 | controller | resource_request: null
2026-03-14 08:54:18.128627 | controller | revision_number: 1
2026-03-14 08:54:18.128774 | controller | qos_network_policy_id: null
2026-03-14 08:54:18.128957 | controller | qos_policy_id: null
2026-03-14 08:54:18.129092 | controller | security_group_ids: []
2026-03-14 08:54:18.129219 | controller | status: DOWN
2026-03-14 08:54:18.129368 | controller | tags: []
2026-03-14 08:54:18.129509 | controller | trunk_details: null
2026-03-14 08:54:18.129642 | controller | trusted: null
2026-03-14 08:54:18.129808 | controller | updated_at: '2026-03-14T08:54:17Z'
2026-03-14 08:54:18.285137 | controller | changed
2026-03-14 08:54:18.299727 | 
2026-03-14 08:54:18.299886 | TASK [Set common facts for further usage]
2026-03-14 08:54:18.385518 | controller | ok
2026-03-14 08:54:18.400310 | 
2026-03-14 08:54:18.400487 | TASK [Create the trunk port]
2026-03-14 08:54:18.951222 | controller | skipping: Conditional result was False
2026-03-14 08:54:18.965466 | 
2026-03-14 08:54:18.965644 | TASK [Avoid NetworkManager to automatically picked up the new interface]
2026-03-14 08:54:19.043611 | controller | skipping: Conditional result was False
2026-03-14 08:54:19.057521 | 
2026-03-14 08:54:19.057698 | TASK [Refresh NetworkManager]
2026-03-14 08:54:19.126347 | controller | skipping: Conditional result was False
2026-03-14 08:54:19.141160 | 
2026-03-14 08:54:19.141339 | TASK [Attach the port to the instance]
2026-03-14 08:54:30.215510 | controller | changed
2026-03-14 08:54:30.229333 | 
2026-03-14 08:54:30.229531 | TASK [Wait till the interface exists]
2026-03-14 08:54:30.576088 | controller -> 38.102.83.32 | changed
2026-03-14 08:54:30.591796 | 
2026-03-14 08:54:30.592000 | TASK [Prepare interface data to be used configuring trunk and vlan ifaces]
2026-03-14 08:54:30.742908 | controller | ok
2026-03-14 08:54:30.755084 | 
2026-03-14 08:54:30.755237 | LOOP [Create VLANs ports and attach them to the trunk]
2026-03-14 08:54:30.817465 | controller | skipping: Conditional result was False
2026-03-14 08:54:33.832737 | controller | skipping: Conditional result was False
2026-03-14 08:54:36.840743 | controller | skipping: Conditional result was False
2026-03-14 08:54:36.851951 | 
2026-03-14 08:54:36.852096 | LOOP [Prepare VLANs interface data to be used configuring]
2026-03-14 08:54:36.957671 | controller | ok:
2026-03-14 08:54:36.957897 | controller | {
2026-03-14 08:54:36.958063 | controller |   "key": "internal-api",
2026-03-14 08:54:36.958096 | controller |   "value": {
2026-03-14 08:54:36.958115 | controller |     "ip": "172.17.0.5"
2026-03-14 08:54:36.958131 | controller |   }
2026-03-14 08:54:36.958148 | controller | }
2026-03-14 08:54:36.958180 | 
2026-03-14 08:54:36.988754 | controller | ok:
2026-03-14 08:54:36.988935 | controller | {
2026-03-14 08:54:36.988966 | controller |   "key": "storage",
2026-03-14 08:54:36.988984 | controller |   "value": {
2026-03-14 08:54:36.989001 | controller |     "ip": "172.18.0.5"
2026-03-14 08:54:36.989036 | controller |   }
2026-03-14 08:54:36.989053 | controller | }
2026-03-14 08:54:37.018842 | controller | ok:
2026-03-14 08:54:37.018904 | controller | {
2026-03-14 08:54:37.018926 | controller |   "key": "tenant",
2026-03-14 08:54:37.018944 | controller |   "value": {
2026-03-14 08:54:37.018960 | controller |     "ip": "172.19.0.5"
2026-03-14 08:54:37.018976 | controller |   }
2026-03-14 08:54:37.018992 | controller | }
2026-03-14 08:54:37.029967 | 
2026-03-14 08:54:37.030101 | LOOP [Create NetworkManager configuration file for the trunk port]
2026-03-14 08:54:37.952364 | controller -> 38.102.83.32 | changed:
2026-03-14 08:54:37.952973 | controller -> 38.102.83.32 | {
2026-03-14 08:54:37.953080 | controller -> 38.102.83.32 |   "key": "default",
2026-03-14 08:54:37.953126 | controller -> 38.102.83.32 |   "value": {
2026-03-14 08:54:37.953163 | controller -> 38.102.83.32 |     "connection": "ci-private-network",
2026-03-14 08:54:37.953200 | controller -> 38.102.83.32 |     "gw": "192.168.122.1",
2026-03-14 08:54:37.953236 | controller -> 38.102.83.32 |     "iface": "ens7",
2026-03-14 08:54:37.953272 | controller -> 38.102.83.32 |     "ip": "192.168.122.10/24",
2026-03-14 08:54:37.953306 | controller -> 38.102.83.32 |     "mac": "fa:16:3e:e3:c1:c5",
2026-03-14 08:54:37.953340 | controller -> 38.102.83.32 |     "mtu": "1500"
2026-03-14 08:54:37.953373 | controller -> 38.102.83.32 |   }
2026-03-14 08:54:37.953408 | controller -> 38.102.83.32 | }
2026-03-14 08:54:38.926383 | controller -> 38.102.83.32 | changed:
2026-03-14 08:54:38.926533 | controller -> 38.102.83.32 | {
2026-03-14 08:54:38.926587 | controller -> 38.102.83.32 |   "key": "internal-api",
2026-03-14 08:54:38.926631 | controller -> 38.102.83.32 |   "value": {
2026-03-14 08:54:38.926671 | controller -> 38.102.83.32 |     "connection": "ci-private-network-20",
2026-03-14 08:54:38.926711 | controller -> 38.102.83.32 |     "iface": "ens7.20",
2026-03-14 08:54:38.926749 | controller -> 38.102.83.32 |     "ip": "172.17.0.5/24",
2026-03-14 08:54:38.926788 | controller -> 38.102.83.32 |     "mac": "52:54:00:3d:3e:99",
2026-03-14 08:54:38.926826 | controller -> 38.102.83.32 |     "mtu": "1496",
2026-03-14 08:54:38.926863 | controller -> 38.102.83.32 |     "parent_iface": "ens7",
2026-03-14 08:54:38.926901 | controller -> 38.102.83.32 |     "vlan": 20
2026-03-14 08:54:38.926940 | controller -> 38.102.83.32 |   }
2026-03-14 08:54:38.926978 | controller -> 38.102.83.32 | }
2026-03-14 08:54:39.892159 | controller -> 38.102.83.32 | changed:
2026-03-14 08:54:39.892514 | controller -> 38.102.83.32 | {
2026-03-14 08:54:39.892621 | controller -> 38.102.83.32 |   "key": "storage",
2026-03-14 08:54:39.892695 | controller -> 38.102.83.32 |   "value": {
2026-03-14 08:54:39.892775 | controller -> 38.102.83.32 |     "connection": "ci-private-network-21",
2026-03-14 08:54:39.892854 | controller -> 38.102.83.32 |     "iface": "ens7.21",
2026-03-14 08:54:39.892935 | controller -> 38.102.83.32 |     "ip": "172.18.0.5/24",
2026-03-14 08:54:39.893010 | controller -> 38.102.83.32 |     "mac": "52:54:00:27:fd:d9",
2026-03-14 08:54:39.893151 | controller -> 38.102.83.32 |     "mtu": "1496",
2026-03-14 08:54:39.893230 | controller -> 38.102.83.32 |     "parent_iface": "ens7",
2026-03-14 08:54:39.893296 | controller -> 38.102.83.32 |     "vlan": 21
2026-03-14 08:54:39.893364 | controller -> 38.102.83.32 |   }
2026-03-14 08:54:39.893441 | controller -> 38.102.83.32 | }
2026-03-14 08:54:40.878946 | controller -> 38.102.83.32 | changed:
2026-03-14 08:54:40.879099 | controller -> 38.102.83.32 | {
2026-03-14 08:54:40.879126 | controller -> 38.102.83.32 |   "key": "tenant",
2026-03-14 08:54:40.879145 | controller -> 38.102.83.32 |   "value": {
2026-03-14 08:54:40.879163 | controller -> 38.102.83.32 |     "connection": "ci-private-network-22",
2026-03-14 08:54:40.879181 | controller -> 38.102.83.32 |     "iface": "ens7.22",
2026-03-14 08:54:40.879199 | controller -> 38.102.83.32 |     "ip": "172.19.0.5/24",
2026-03-14 08:54:40.879215 | controller -> 38.102.83.32 |     "mac": "52:54:00:6e:7d:ad",
2026-03-14 08:54:40.879232 | controller -> 38.102.83.32 |     "mtu": "1496",
2026-03-14 08:54:40.879248 | controller -> 38.102.83.32 |     "parent_iface": "ens7",
2026-03-14 08:54:40.879263 | controller -> 38.102.83.32 |     "vlan": 22
2026-03-14 08:54:40.879280 | controller -> 38.102.83.32 |   }
2026-03-14 08:54:40.879296 | controller -> 38.102.83.32 | }
2026-03-14 08:54:40.905578 | 
2026-03-14 08:54:40.905785 | TASK [Refresh NetworkManager]
2026-03-14 08:54:41.822660 | controller -> 38.102.83.32 | changed
2026-03-14 08:54:41.842741 | 
2026-03-14 08:54:41.842896 | TASK [Debug fetch IP routes]
2026-03-14 08:54:42.191143 | controller -> 38.102.83.32 | changed
2026-03-14 08:54:42.207850 | 
2026-03-14 08:54:42.208083 | TASK [Debug IP routes]
2026-03-14 08:54:42.268002 | controller -> 38.102.83.32 | ok
2026-03-14 08:54:42.285579 | 
2026-03-14 08:54:42.285803 | TASK [Display some data about network ports]
2026-03-14 08:54:44.598991 | controller | +--------------------------------------+-------------------------------------------------+-------------------+-------------------------------------------------------------------------------+--------+
2026-03-14 08:54:44.702462 | controller | | ID                                   | Name                                            | MAC Address       | Fixed IP Addresses                                                            | Status |
2026-03-14 08:54:44.702562 | controller | +--------------------------------------+-------------------------------------------------+-------------------+-------------------------------------------------------------------------------+--------+
2026-03-14 08:54:44.702569 | controller | | 022709b1-dd59-437e-b00d-049cb0e9bf02 | crc-6518bcde-aa50-4603-92c7-71dcf31294f9        | fa:16:3e:e3:c1:c5 | ip_address='192.168.122.10', subnet_id='a281b98f-17e4-4b27-b77d-2827e0a63895' | ACTIVE |
2026-03-14 08:54:44.702574 | controller | | 3fb3d0b3-3be6-4ea1-b59d-3121f46e758b | controller-a1239aa3-61dc-4384-a93b-9c354d165e80 | fa:16:3e:ac:8d:09 | ip_address='192.168.122.11', subnet_id='a281b98f-17e4-4b27-b77d-2827e0a63895' | ACTIVE |
2026-03-14 08:54:44.702578 | controller | | eb54d78c-1ed8-4858-b9b0-dae591a66717 |                                                 | fa:16:3e:0d:e4:5a |                                                                               | DOWN   |
2026-03-14 08:54:44.702584 | controller | +--------------------------------------+-------------------------------------------------+-------------------+-------------------------------------------------------------------------------+--------+
2026-03-14 08:54:44.861501 | controller | changed
2026-03-14 08:54:44.876195 | 
2026-03-14 08:54:44.876378 | LOOP [Display server configuration]
2026-03-14 08:54:47.150060 | controller | changed
2026-03-14 08:54:47.150526 | controller | changed: All items complete
2026-03-14 08:54:47.150605 | 
2026-03-14 08:54:51.979215 | controller | changed
2026-03-14 08:54:51.997000 | 
2026-03-14 08:54:51.997196 | TASK [Fetch underneath provider DNSs]
2026-03-14 08:54:52.028740 | controller | ok
2026-03-14 08:54:52.043199 | controller | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-fetch-provider-dns.yml
2026-03-14 08:54:52.058087 | 
2026-03-14 08:54:52.058187 | TASK [Fetch the DNS list of the default interface]
2026-03-14 08:54:52.389379 | controller | ok
2026-03-14 08:54:52.405923 | 
2026-03-14 08:54:52.406224 | TASK [Set provider DNS for further usage]
2026-03-14 08:54:52.494500 | controller | ok
2026-03-14 08:54:52.510906 | 
2026-03-14 08:54:52.511197 | LOOP [Save networking data to file for further usage]
2026-03-14 08:54:53.440144 | controller -> 38.102.83.32 | changed: "crc"
2026-03-14 08:54:56.771110 | controller -> 38.102.83.180 | changed: "controller"
2026-03-14 08:54:56.779677 | 
2026-03-14 08:54:56.779796 | TASK [Remove cloud_secrets file]
2026-03-14 08:54:59.801275 | controller | changed
2026-03-14 08:54:59.837857 | 
2026-03-14 08:54:59.838072 | PLAY RECAP
2026-03-14 08:54:59.838143 | controller | ok: 48 changed: 22 unreachable: 0 failed: 0 skipped: 14 rescued: 0 ignored: 0
2026-03-14 08:54:59.838166 | 
2026-03-14 08:55:00.003954 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network.yml@master]
2026-03-14 08:55:00.008601 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/prepare-crc.yaml@master]
2026-03-14 08:55:00.701629 | 
2026-03-14 08:55:00.701782 | PLAY [Prefer ipv4 over ipv6]
2026-03-14 08:55:00.738301 | 
2026-03-14 08:55:00.738425 | TASK [Configure /etc/gai.conf to prever ipv4 over ipv6]
2026-03-14 08:55:00.784838 | crc | skipping: Conditional result was False
2026-03-14 08:55:00.795665 | controller | skipping: Conditional result was False
2026-03-14 08:55:00.843340 | 
2026-03-14 08:55:00.843459 | PLAY [Generic task for migration time to crc-cloud]
2026-03-14 08:55:00.865478 | 
2026-03-14 08:55:00.865588 | TASK [Check if there is a image build date file]
2026-03-14 08:55:01.327112 | crc | ok
2026-03-14 08:55:01.341937 | 
2026-03-14 08:55:01.342140 | TASK [Print image date]
2026-03-14 08:55:01.803063 | crc | Sat Mar 14 02:51:45 UTC 2026 | Image type: crc-cloud
2026-03-14 08:55:01.931601 | crc | ok
2026-03-14 08:55:01.947559 | 
2026-03-14 08:55:01.947849 | TASK [Run crc-cloud preparation]
2026-03-14 08:55:01.980843 | crc | ok
2026-03-14 08:55:02.028295 | 
2026-03-14 08:55:02.028452 | TASK [prepare-crc-cloud : Print build date]
2026-03-14 08:55:02.055394 | crc | ok
2026-03-14 08:55:02.069421 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/build_date.yaml
2026-03-14 08:55:02.084358 | 
2026-03-14 08:55:02.084504 | TASK [prepare-crc-cloud : Check if there is a image build date file]
2026-03-14 08:55:02.372512 | crc | ok
2026-03-14 08:55:02.391152 | 
2026-03-14 08:55:02.391334 | TASK [prepare-crc-cloud : Print image date if available]
2026-03-14 08:55:02.719585 | crc | Sat Mar 14 02:51:45 UTC 2026 | Image type: crc-cloud
2026-03-14 08:55:02.949129 | crc | ok
2026-03-14 08:55:02.966252 | 
2026-03-14 08:55:02.966414 | TASK [prepare-crc-cloud : Tune Kubelet config file]
2026-03-14 08:55:02.993554 | crc | skipping: Conditional result was False
2026-03-14 08:55:03.011612 | 
2026-03-14 08:55:03.011803 | TASK [prepare-crc-cloud : Reboot CRC host to get the network interfaces]
2026-03-14 08:55:03.057319 | crc | ok
2026-03-14 08:55:03.075898 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/reboot_crc.yaml
2026-03-14 08:55:03.091515 | 
2026-03-14 08:55:03.091621 | TASK [prepare-crc-cloud : Reboot host after adding pull secret and changing kubelet node params]
2026-03-14 08:55:58.183920 | crc | changed
2026-03-14 08:55:58.199595 | 
2026-03-14 08:55:58.199772 | TASK [Start Zuul console after reboot]
2026-03-14 08:55:58.235741 | crc | ok
2026-03-14 08:55:58.265485 | 
2026-03-14 08:55:58.265625 | TASK [start-zuul-console : Start zuul_console daemon.]
2026-03-14 08:55:58.886743 | crc | ok
2026-03-14 08:55:58.916827 | 
2026-03-14 08:55:58.917048 | TASK [prepare-crc-cloud : Prepare for executing crc-cloud]
2026-03-14 08:55:58.953219 | crc | ok
2026-03-14 08:55:58.967425 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/pre.yaml
2026-03-14 08:55:58.984603 | 
2026-03-14 08:55:58.984739 | TASK [prepare-crc-cloud : Disable overwriting /etc/resolv.conf by the NetworkManager]
2026-03-14 08:56:00.139976 | crc | changed
2026-03-14 08:56:00.180101 | 
2026-03-14 08:56:00.180218 | TASK [prepare-crc-cloud : Reload NetworkManager after creating custom rules]
2026-03-14 08:56:01.062059 | crc | changed
2026-03-14 08:56:01.081058 | 
2026-03-14 08:56:01.081177 | TASK [prepare-crc-cloud : Inject pull secret into /var/lib/kubelet/config.json]
2026-03-14 08:56:02.063510 | crc | changed
2026-03-14 08:56:02.083591 | 
2026-03-14 08:56:02.083779 | TASK [prepare-crc-cloud : Check if etcd slow profile set]
2026-03-14 08:56:02.110850 | crc | skipping: Conditional result was False
2026-03-14 08:56:02.124130 | 
2026-03-14 08:56:02.124261 | TASK [prepare-crc-cloud : Use ramdisk for etcd]
2026-03-14 08:56:02.159606 | crc | ok
2026-03-14 08:56:02.176667 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/ramdisk.yaml
2026-03-14 08:56:02.195633 | 
2026-03-14 08:56:02.195783 | TASK [prepare-crc-cloud : Move current etcd to other place]
2026-03-14 08:56:02.785081 | crc | ok
2026-03-14 08:56:02.795647 | 
2026-03-14 08:56:02.795773 | TASK [prepare-crc-cloud : Mount ramdisk]
2026-03-14 08:56:03.278088 | crc | changed
2026-03-14 08:56:03.293799 | 
2026-03-14 08:56:03.294063 | TASK [prepare-crc-cloud : Set proper permissions after mount]
2026-03-14 08:56:03.776412 | crc | changed
2026-03-14 08:56:03.793958 | 
2026-03-14 08:56:03.794177 | TASK [prepare-crc-cloud : Copy content from old location to ramdisk]
2026-03-14 08:56:05.292418 | crc | changed
2026-03-14 08:56:05.301913 | 
2026-03-14 08:56:05.302067 | TASK [prepare-crc-cloud : Set proper SELinux context]
2026-03-14 08:56:05.851813 | crc | ok
2026-03-14 08:56:05.868612 | 
2026-03-14 08:56:05.868801 | TASK [prepare-crc-cloud : Block crc.testing domains]
2026-03-14 08:56:05.913254 | crc | ok
2026-03-14 08:56:05.934873 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/block_crc_domains.yaml
2026-03-14 08:56:05.962771 | 
2026-03-14 08:56:05.962915 | LOOP [prepare-crc-cloud : Set crc.testing domains into /etc/hosts]
2026-03-14 08:56:06.425243 | crc | changed: line added
2026-03-14 08:56:06.684716 | crc | changed: line added
2026-03-14 08:56:06.973677 | crc | changed: line added
2026-03-14 08:56:07.277367 | crc | changed: line added
2026-03-14 08:56:07.602723 | crc | changed: line added
2026-03-14 08:56:07.822678 | crc | changed: line added
2026-03-14 08:56:08.118472 | crc | changed: line added
2026-03-14 08:56:08.375808 | crc | changed: line added
2026-03-14 08:56:08.400231 | 
2026-03-14 08:56:08.400591 | TASK [prepare-crc-cloud : Get default interface name]
2026-03-14 08:56:08.723305 | crc | ens3
2026-03-14 08:56:08.964894 | crc | ok
2026-03-14 08:56:08.982195 | 
2026-03-14 08:56:08.982696 | LOOP [prepare-crc-cloud : Block CRC domains also using iptables]
2026-03-14 08:56:09.530102 | crc | ok
2026-03-14 08:56:09.530319 | crc | changed: All items complete
2026-03-14 08:56:09.530349 | 
2026-03-14 08:56:09.854822 | crc | ok
2026-03-14 08:56:10.174130 | crc | ok
2026-03-14 08:56:10.510557 | crc | ok
2026-03-14 08:56:10.786372 | crc | ok
2026-03-14 08:56:11.083915 | crc | ok
2026-03-14 08:56:11.386755 | crc | ok
2026-03-14 08:56:11.697490 | crc | ok
2026-03-14 08:56:11.730522 | 
2026-03-14 08:56:11.730744 | TASK [prepare-crc-cloud : Set required facts]
2026-03-14 08:56:11.775442 | crc | ok
2026-03-14 08:56:11.795231 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/set_vars.yaml
2026-03-14 08:56:11.817772 | 
2026-03-14 08:56:11.817898 | TASK [prepare-crc-cloud : Set required facts]
2026-03-14 08:56:11.905538 | crc | Output suppressed because no_log was given
2026-03-14 08:56:11.915523 | 
2026-03-14 08:56:11.915619 | TASK [Start crc-cloud role when Zuul defined]
2026-03-14 08:56:11.962372 | crc | ok
2026-03-14 08:56:12.030712 | 
2026-03-14 08:56:12.030851 | TASK [deploy-crc-cloud : Check if pull-secret is provided]
2026-03-14 08:56:12.055088 | crc | skipping: Conditional result was False
2026-03-14 08:56:12.063036 | 
2026-03-14 08:56:12.063110 | TASK [deploy-crc-cloud : Create kubeconfig]
2026-03-14 08:56:12.096570 | crc | ok
2026-03-14 08:56:12.109249 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/kubeconfig.yaml
2026-03-14 08:56:12.122398 | 
2026-03-14 08:56:12.122476 | TASK [deploy-crc-cloud : Create kube directory]
2026-03-14 08:56:12.416622 | crc | changed
2026-03-14 08:56:12.432460 | 
2026-03-14 08:56:12.432605 | TASK [deploy-crc-cloud : Copy kubeconfig to user dir]
2026-03-14 08:56:12.970193 | crc | changed
2026-03-14 08:56:12.980989 | 
2026-03-14 08:56:12.981162 | TASK [deploy-crc-cloud : Setup dnsmasq]
2026-03-14 08:56:13.008386 | crc | ok
2026-03-14 08:56:13.024596 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/dnsmasq.yaml
2026-03-14 08:56:13.044309 | 
2026-03-14 08:56:13.044449 | TASK [deploy-crc-cloud : Disable overwriting /etc/resolv.conf by the NetworkManager]
2026-03-14 08:56:13.905694 | crc | ok
2026-03-14 08:56:13.916214 | 
2026-03-14 08:56:13.916313 | TASK [deploy-crc-cloud : Restart NetworkManager when its needed]
2026-03-14 08:56:13.941388 | crc | skipping: Conditional result was False
2026-03-14 08:56:13.950634 | 
2026-03-14 08:56:13.950714 | TASK [deploy-crc-cloud : Ensure /etc/resolv.conf is not a symlink and is writable]
2026-03-14 08:56:14.286738 | crc | ok
2026-03-14 08:56:14.302224 | 
2026-03-14 08:56:14.302364 | TASK [deploy-crc-cloud : Create crc-dnsmasq.conf]
2026-03-14 08:56:15.268642 | crc | changed
2026-03-14 08:56:15.278557 | 
2026-03-14 08:56:15.278645 | TASK [deploy-crc-cloud : Add domains into crc-dnsmasq.conf when alternative_domain]
2026-03-14 08:56:15.303784 | crc | skipping: Conditional result was False
2026-03-14 08:56:15.322008 | 
2026-03-14 08:56:15.322211 | LOOP [deploy-crc-cloud : Set this host as first nameserver in /etc/resolv.conf]
2026-03-14 08:56:15.723227 | crc | changed: line replaced
2026-03-14 08:56:16.001810 | crc | ok
2026-03-14 08:56:16.895901 | crc | ok
2026-03-14 08:56:16.912455 | 
2026-03-14 08:56:16.912576 | TASK [deploy-crc-cloud : Restart dnsmasq]
2026-03-14 08:56:19.054229 | crc | changed
2026-03-14 08:56:19.074473 | 
2026-03-14 08:56:19.074642 | TASK [deploy-crc-cloud : Start kubelet]
2026-03-14 08:56:19.102424 | crc | ok
2026-03-14 08:56:19.116323 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/kubelet.yaml
2026-03-14 08:56:19.140714 | 
2026-03-14 08:56:19.140865 | TASK [deploy-crc-cloud : Start and enable kubelet]
2026-03-14 08:56:35.219952 | crc | changed
2026-03-14 08:56:35.240202 | 
2026-03-14 08:56:35.240378 | TASK [deploy-crc-cloud : Wait for port 6443 to be up]
2026-03-14 08:57:35.729259 | crc | ok
2026-03-14 08:57:35.748743 | 
2026-03-14 08:57:35.748930 | TASK [deploy-crc-cloud : Wait for API to start before continue]
2026-03-14 08:57:37.330014 | crc | NAMESPACE                         NAME                                         READY   STATUS    RESTARTS   AGE
2026-03-14 08:57:37.330137 | crc | openshift-network-console         networking-console-plugin-85b44fc459-gdk6g   1/1     Running   2          384d
2026-03-14 08:57:37.330149 | crc | openshift-network-diagnostics     network-check-source-55646444c4-trplf        1/1     Running   2          384d
2026-03-14 08:57:37.330157 | crc | openshift-network-diagnostics     network-check-target-xd92c                   1/1     Running   2          384d
2026-03-14 08:57:37.330174 | crc | openshift-network-node-identity   network-node-identity-vrzqb                  2/2     Running   7          384d
2026-03-14 08:57:37.340258 | crc | openshift-network-operator        iptables-alerter-4ln5h                       1/1     Running   2          384d
2026-03-14 08:57:37.340289 | crc | openshift-network-operator        network-operator-58b4c7f79c-55gtf            1/1     Running   4          384d
2026-03-14 08:57:37.813504 | crc | ok
2026-03-14 08:57:37.831500 | 
2026-03-14 08:57:37.831695 | TASK [deploy-crc-cloud : Replace default pubkey]
2026-03-14 08:57:37.874700 | crc | ok
2026-03-14 08:57:37.894980 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/pubkey.yaml
2026-03-14 08:57:37.921702 | 
2026-03-14 08:57:37.921848 | TASK [deploy-crc-cloud : Check if id_rsa.pub exists]
2026-03-14 08:57:38.219724 | crc | ok
2026-03-14 08:57:38.238957 | 
2026-03-14 08:57:38.239173 | TASK [deploy-crc-cloud : Read pub key]
2026-03-14 08:57:38.793377 | crc | skipping: Conditional result was False
2026-03-14 08:57:38.811749 | 
2026-03-14 08:57:38.811934 | TASK [deploy-crc-cloud : Wait for machineconfig]
2026-03-14 08:57:38.838847 | crc | skipping: Conditional result was False
2026-03-14 08:57:38.855559 | 
2026-03-14 08:57:38.855736 | TASK [deploy-crc-cloud : Patch machineconfig 99-master-ssh]
2026-03-14 08:57:39.449279 | crc | skipping: Conditional result was False
2026-03-14 08:57:39.466736 | 
2026-03-14 08:57:39.466882 | TASK [deploy-crc-cloud : Accept certificate]
2026-03-14 08:57:39.549864 | crc | ok
2026-03-14 08:57:39.569773 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/accept_cert.yaml
2026-03-14 08:57:39.607753 | 
2026-03-14 08:57:39.607876 | LOOP [deploy-crc-cloud : Re-run accept certificates in Pending state few times]
2026-03-14 08:57:40.122359 | crc | ok
2026-03-14 08:57:40.122596 | crc | changed: All items complete
2026-03-14 08:57:40.122633 | 
2026-03-14 08:57:50.752846 | crc | ok
2026-03-14 08:58:01.196159 | crc | ok
2026-03-14 08:58:01.218843 | 
2026-03-14 08:58:01.219060 | TASK [deploy-crc-cloud : Create service account for - auto-csr-approver]
2026-03-14 08:58:02.041609 | crc | changed
2026-03-14 08:58:02.049501 | 
2026-03-14 08:58:02.049570 | TASK [deploy-crc-cloud : Apply cluster cert approver]
2026-03-14 08:58:02.524387 | crc | serviceaccount/csr-approver-sa created
2026-03-14 08:58:02.528692 | crc | clusterrole.rbac.authorization.k8s.io/csr-approver created
2026-03-14 08:58:02.535269 | crc | clusterrolebinding.rbac.authorization.k8s.io/csr-approver-binding created
2026-03-14 08:58:02.542956 | crc | cronjob.batch/auto-csr-approver created
2026-03-14 08:58:02.602247 | crc | ok
2026-03-14 08:58:02.618965 | 
2026-03-14 08:58:02.619180 | TASK [deploy-crc-cloud : Wait for cluster become healthy]
2026-03-14 08:58:02.664457 | crc | ok
2026-03-14 08:58:02.683940 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/wait_cluster_become_healthy.yaml
2026-03-14 08:58:02.720867 | 
2026-03-14 08:58:02.721059 | TASK [deploy-crc-cloud : Check if specified OpenShift components are healthy etcd|openshift-apiserver]
2026-03-14 08:58:03.199087 | crc | True
2026-03-14 08:58:03.199893 | crc | False
2026-03-14 08:59:51.349617 | crc | changed
2026-03-14 08:59:51.365833 | 
2026-03-14 08:59:51.366010 | TASK [deploy-crc-cloud : Output success message if components are healthy]
2026-03-14 08:59:51.408573 | crc | ok: OpenShift cluster components have become healthy in approximately 60 * 15 seconds.
2026-03-14 08:59:51.427077 | 
2026-03-14 08:59:51.427282 | TASK [deploy-crc-cloud : Fail if any components are still unhealthy]
2026-03-14 08:59:51.444732 | crc | skipping: Conditional result was False
2026-03-14 08:59:51.458955 | 
2026-03-14 08:59:51.459195 | TASK [deploy-crc-cloud : Set credentials]
2026-03-14 08:59:51.496114 | crc | ok
2026-03-14 08:59:51.510280 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/set_credentials.yaml
2026-03-14 08:59:51.534864 | 
2026-03-14 08:59:51.534993 | TASK [deploy-crc-cloud : Ensure htpasswd.txt does not exists]
2026-03-14 08:59:51.849371 | crc | ok
2026-03-14 08:59:51.865352 | 
2026-03-14 08:59:51.865490 | TASK [deploy-crc-cloud : Create temporary directory]
2026-03-14 08:59:52.266605 | crc | changed
2026-03-14 08:59:52.282394 | 
2026-03-14 08:59:52.282530 | LOOP [deploy-crc-cloud : Get htpasswd]
2026-03-14 08:59:52.368188 | crc | Output suppressed because no_log was given
2026-03-14 08:59:52.387109 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/get_htpasswd.yaml
2026-03-14 08:59:52.389057 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/get_htpasswd.yaml
2026-03-14 08:59:52.391073 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/get_htpasswd.yaml
2026-03-14 08:59:52.414347 | 
2026-03-14 08:59:52.414476 | TASK [deploy-crc-cloud : Get htpasswd for developer]
2026-03-14 09:00:18.632786 | crc | time="2026-03-14T09:00:18Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
2026-03-14 09:00:18.637730 | crc | Trying to pull ghcr.io/crc-org/crc-cloud/httpd-tools:latest...
2026-03-14 09:00:19.862606 | crc | Getting image source signatures
2026-03-14 09:00:19.862798 | crc | Copying blob sha256:1730cf1a00b8c06596486f3456376e7660a42976bb9fcd6604ae22a70fca6b65
2026-03-14 09:00:19.863076 | crc | Copying blob sha256:f391d299bb65afd981883c104f434a04ee3ffb247e341247171779ef526ed65f
2026-03-14 09:00:23.729900 | crc | Copying config sha256:d81064f8940a06af9e1a39fc510c6bfc299bb166f99ee6a892f22ec84ee58041
2026-03-14 09:00:23.756557 | crc | Writing manifest to image destination
2026-03-14 09:00:25.045834 | crc | changed
2026-03-14 09:00:25.057748 | 
2026-03-14 09:00:25.057890 | TASK [deploy-crc-cloud : Get htpasswd for kubeadmin]
2026-03-14 09:00:25.484611 | crc | time="2026-03-14T09:00:25Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
2026-03-14 09:00:26.102355 | crc | changed
2026-03-14 09:00:26.120955 | 
2026-03-14 09:00:26.121174 | TASK [deploy-crc-cloud : Get htpasswd for redhat]
2026-03-14 09:00:26.512220 | crc | time="2026-03-14T09:00:26Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
2026-03-14 09:00:27.184563 | crc | changed
2026-03-14 09:00:27.203535 | 
2026-03-14 09:00:27.203750 | TASK [deploy-crc-cloud : Cleanup htpasswd.txt file]
2026-03-14 09:00:27.771833 | crc | changed
2026-03-14 09:00:27.779655 | 
2026-03-14 09:00:27.779728 | TASK [deploy-crc-cloud : Create secret with generic htpass-secret]
2026-03-14 09:00:28.352138 | crc | ok
2026-03-14 09:00:28.368217 | 
2026-03-14 09:00:28.368380 | TASK [deploy-crc-cloud : Replace htpass-secret]
2026-03-14 09:00:28.900651 | crc | secret/htpass-secret replaced
2026-03-14 09:00:28.975492 | crc | ok
2026-03-14 09:00:28.984035 | 
2026-03-14 09:00:28.984243 | TASK [deploy-crc-cloud : Remove temporary directory]
2026-03-14 09:00:29.321792 | crc | changed
2026-03-14 09:00:29.340223 | 
2026-03-14 09:00:29.340419 | TASK [deploy-crc-cloud : Replace default CA]
2026-03-14 09:00:29.376422 | crc | ok
2026-03-14 09:00:29.394576 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/replace_default_ca.yaml
2026-03-14 09:00:29.430044 | 
2026-03-14 09:00:29.430139 | TASK [deploy-crc-cloud : Generate key]
2026-03-14 09:00:32.983456 | crc | changed
2026-03-14 09:00:33.000256 | 
2026-03-14 09:00:33.000571 | TASK [deploy-crc-cloud : Generate cert]
2026-03-14 09:00:33.572920 | crc | changed
2026-03-14 09:00:33.580440 | 
2026-03-14 09:00:33.580509 | TASK [deploy-crc-cloud : Generate csr]
2026-03-14 09:00:33.912439 | crc | ......+...+.+.....+.......+..+.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.................+.+...+..+.+.....+...............+.......+.....+.+..+....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+...+.....+.......+........................+..+....+...+..+...+.............+...+.....+.......+.........+......+...+...........+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2026-03-14 09:00:34.105073 | crc | .........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+.+.....+......+...+.+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+....+...+...............+...+...+..+....+...+......+..+...+....+...+...+.....+.+...+..+...+.+....................+....+..+...+............+................+....................+.......+...+..+.......+...+...+..................+..+...+.........+...+.......+........+...+.......+..+...+..................+.+...+.....+...+...+..........+..............+...+..................+....+......+...........+.+..+...+.......+.....+.+.....+...............+.....................+.+..+.+..+............+..........+.................+.+............+..+......+...+....+.....+.+......+.....+...+.+...........+.+.....+...+....+...........+.......+.....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2026-03-14 09:00:34.108391 | crc | -----
2026-03-14 09:00:34.159322 | crc | changed
2026-03-14 09:00:34.175517 | 
2026-03-14 09:00:34.175717 | TASK [deploy-crc-cloud : Generate user cert]
2026-03-14 09:00:34.605241 | crc | Certificate request self-signature ok
2026-03-14 09:00:34.613524 | crc | subject=O = ${GROUP}, CN = core
2026-03-14 09:00:34.796440 | crc | changed
2026-03-14 09:00:34.804006 | 
2026-03-14 09:00:34.804098 | TASK [deploy-crc-cloud : Create configmap]
2026-03-14 09:00:35.336584 | crc | configmap/client-ca-custom created
2026-03-14 09:00:35.425177 | crc | ok
2026-03-14 09:00:35.444834 | 
2026-03-14 09:00:35.445065 | TASK [deploy-crc-cloud : Patch apiserver]
2026-03-14 09:00:36.013571 | crc | apiserver.config.openshift.io/cluster patched
2026-03-14 09:00:36.522405 | crc | ok
2026-03-14 09:00:36.539004 | 
2026-03-14 09:00:36.539200 | TASK [deploy-crc-cloud : Create configmap]
2026-03-14 09:00:37.015971 | crc | W0314 09:00:37.015868   13965 helpers.go:703] --dry-run is deprecated and can be replaced with --dry-run=client.
2026-03-14 09:00:37.058521 | crc | configmap/admin-kubeconfig-client-ca replaced
2026-03-14 09:00:37.131994 | crc | ok
2026-03-14 09:00:37.141533 | 
2026-03-14 09:00:37.141639 | TASK [deploy-crc-cloud : Login to the OpenShift cluster]
2026-03-14 09:00:37.184437 | crc | ok
2026-03-14 09:00:37.198345 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/login.yaml
2026-03-14 09:00:37.241093 | 
2026-03-14 09:00:37.241230 | TASK [deploy-crc-cloud : Try to login after all changes]
2026-03-14 09:00:37.661369 | crc | WARNING: Using insecure TLS client config. Setting this option is not supported!
2026-03-14 09:00:37.826680 | crc |
2026-03-14 09:00:37.826741 | crc | Login failed (401 Unauthorized)
2026-03-14 09:00:37.833126 | crc | Verify you have provided the correct credentials.
2026-03-14 09:01:55.117189 | crc | ok
2026-03-14 09:01:55.127130 | 
2026-03-14 09:01:55.127222 | TASK [deploy-crc-cloud : Patch pull secret]
2026-03-14 09:01:55.153496 | crc | ok
2026-03-14 09:01:55.163983 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/patch_pull_secret.yaml
2026-03-14 09:01:55.193174 | 
2026-03-14 09:01:55.193287 | TASK [deploy-crc-cloud : Patch pull secret]
2026-03-14 09:01:55.757758 | crc | Output suppressed because no_log was given
2026-03-14 09:01:55.775119 | 
2026-03-14 09:01:55.775279 | TASK [deploy-crc-cloud : Wait for cluster become healthy after patching CA and pull secret]
2026-03-14 09:01:55.817638 | crc | ok
2026-03-14 09:01:55.835934 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/wait_cluster_become_healthy.yaml
2026-03-14 09:01:55.878188 | 
2026-03-14 09:01:55.878287 | TASK [deploy-crc-cloud : Check if specified OpenShift components are healthy etcd|openshift-apiserver]
2026-03-14 09:01:56.348681 | crc | True
2026-03-14 09:01:56.350036 | crc | True
2026-03-14 09:01:56.417680 | crc | changed
2026-03-14 09:01:56.437488 | 
2026-03-14 09:01:56.437688 | TASK [deploy-crc-cloud : Output success message if components are healthy]
2026-03-14 09:01:56.484706 | crc | ok: OpenShift cluster components have become healthy in approximately 60 * 15 seconds.
2026-03-14 09:01:56.499690 | 
2026-03-14 09:01:56.499851 | TASK [deploy-crc-cloud : Fail if any components are still unhealthy]
2026-03-14 09:01:56.515840 | crc | skipping: Conditional result was False
2026-03-14 09:01:56.530326 | 
2026-03-14 09:01:56.530477 | TASK [deploy-crc-cloud : Create certificate and patch secret]
2026-03-14 09:01:56.545916 | crc | skipping: Conditional result was False
2026-03-14 09:01:56.557499 | 
2026-03-14 09:01:56.557592 | TASK [deploy-crc-cloud : Wait for cluster become healthy after adding domain]
2026-03-14 09:01:56.631494 | crc | ok
2026-03-14 09:01:56.639507 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/wait_cluster_become_healthy.yaml
2026-03-14 09:01:56.673691 | 
2026-03-14 09:01:56.673828 | TASK [deploy-crc-cloud : Check if specified OpenShift components are healthy etcd|openshift-apiserver]
2026-03-14 09:01:57.122293 | crc | True
2026-03-14 09:01:57.123212 | crc | True
2026-03-14 09:01:57.222651 | crc | changed
2026-03-14 09:01:57.239004 | 
2026-03-14 09:01:57.239227 | TASK [deploy-crc-cloud : Output success message if components are healthy]
2026-03-14 09:01:57.291637 | crc | ok: OpenShift cluster components have become healthy in approximately 60 * 15 seconds.
2026-03-14 09:01:57.307836 | 
2026-03-14 09:01:57.307998 | TASK [deploy-crc-cloud : Fail if any components are still unhealthy]
2026-03-14 09:01:57.334645 | crc | skipping: Conditional result was False
2026-03-14 09:01:57.347435 | 
2026-03-14 09:01:57.347572 | TASK [deploy-crc-cloud : Patch ingress config]
2026-03-14 09:01:57.373821 | crc | skipping: Conditional result was False
2026-03-14 09:01:57.392477 | 
2026-03-14 09:01:57.392669 | TASK [deploy-crc-cloud : Patch api server]
2026-03-14 09:01:57.421247 | crc | skipping: Conditional result was False
2026-03-14 09:01:57.441702 | 
2026-03-14 09:01:57.441886 | TASK [deploy-crc-cloud : Patch default route]
2026-03-14 09:01:57.469267 | crc | skipping: Conditional result was False
2026-03-14 09:01:57.487779 | 
2026-03-14 09:01:57.487973 | TASK [deploy-crc-cloud : Wait for cluster become healthy after changing ingress api server and default route]
2026-03-14 09:01:57.523606 | crc | ok
2026-03-14 09:01:57.542896 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/wait_cluster_become_healthy.yaml
2026-03-14 09:01:57.591662 | 
2026-03-14 09:01:57.591773 | TASK [deploy-crc-cloud : Check if specified OpenShift components are healthy authentication|console|etcd|ingress|openshift-apiserver]
2026-03-14 09:01:58.079563 | crc | False
2026-03-14 09:01:58.080703 | crc | True
2026-03-14 09:01:58.080719 | crc | True
2026-03-14 09:01:58.080725 | crc | True
2026-03-14 09:01:58.080731 | crc | True
2026-03-14 09:02:28.954826 | crc | changed
2026-03-14 09:02:28.971529 | 
2026-03-14 09:02:28.971712 | TASK [deploy-crc-cloud : Output success message if components are healthy]
2026-03-14 09:02:29.025437 | crc | ok: OpenShift cluster components have become healthy in approximately 60 * 15 seconds.
2026-03-14 09:02:29.042226 | 
2026-03-14 09:02:29.042370 | TASK [deploy-crc-cloud : Fail if any components are still unhealthy]
2026-03-14 09:02:29.069114 | crc | skipping: Conditional result was False
2026-03-14 09:02:29.090732 | 
2026-03-14 09:02:29.091101 | TASK [deploy-crc-cloud : Get console route]
2026-03-14 09:02:29.118352 | crc | skipping: Conditional result was False
2026-03-14 09:02:29.178907 | 
2026-03-14 09:02:29.179040 | TASK [Start crc-cloud role when Zuul is not defined]
2026-03-14 09:02:29.222954 | crc | skipping: Conditional result was False
2026-03-14 09:02:29.230626 | 
2026-03-14 09:02:29.230704 | TASK [prepare-crc-cloud : Recreate kubelet config.json]
2026-03-14 09:02:29.270130 | crc | ok
2026-03-14 09:02:29.287436 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/inject-pull-secret.yaml
2026-03-14 09:02:29.318134 | 
2026-03-14 09:02:29.318228 | TASK [prepare-crc-cloud : Fail when openshift_pull_sec not provided]
2026-03-14 09:02:29.383117 | crc | skipping: Conditional result was False
2026-03-14 09:02:29.396593 | 
2026-03-14 09:02:29.396726 | TASK [prepare-crc-cloud : Cleanup file]
2026-03-14 09:02:29.949183 | crc | Output suppressed because no_log was given
2026-03-14 09:02:29.964459 | 
2026-03-14 09:02:29.964603 | TASK [prepare-crc-cloud : Copy pull-secret to kubelet config.json]
2026-03-14 09:02:30.343688 | crc | Output suppressed because no_log was given
2026-03-14 09:02:30.360955 | 
2026-03-14 09:02:30.361291 | TASK [prepare-crc-cloud : Replace singleqoute with doublequote]
2026-03-14 09:02:30.879097 | crc | changed: 42 replacements made
2026-03-14 09:02:30.887190 | 
2026-03-14 09:02:30.887298 | TASK [prepare-crc-cloud : Remove spaces]
2026-03-14 09:02:31.299247 | crc | changed: 20 replacements made
2026-03-14 09:02:31.314498 | 
2026-03-14 09:02:31.314632 | TASK [prepare-crc-cloud : Ensure you are logged in to the OpenShift]
2026-03-14 09:02:31.358744 | crc | ok
2026-03-14 09:02:31.377351 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/openshift_login.yaml
2026-03-14 09:02:31.416781 | 
2026-03-14 09:02:31.416871 | TASK [prepare-crc-cloud : Login to the OpenShift]
2026-03-14 09:02:31.927320 | crc | WARNING: Using insecure TLS client config. Setting this option is not supported!
2026-03-14 09:02:32.072336 | crc |
2026-03-14 09:02:32.072404 | crc | Login successful.
2026-03-14 09:02:32.089685 | crc |
2026-03-14 09:02:32.089739 | crc | You have access to 65 projects, the list has been suppressed. You can list all projects with 'oc projects'
2026-03-14 09:02:32.111079 | crc |
2026-03-14 09:02:32.111128 | crc | Using project "default".
2026-03-14 09:02:32.459973 | crc | ok
2026-03-14 09:02:32.467398 | 
2026-03-14 09:02:32.467462 | TASK [prepare-crc-cloud : Post tasks after crc-cloud script execution]
2026-03-14 09:02:32.499944 | crc | ok
2026-03-14 09:02:32.510909 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/post.yaml
2026-03-14 09:02:32.534234 | 
2026-03-14 09:02:32.534331 | LOOP [prepare-crc-cloud : Set DNS nameservers provided by cloud provider]
2026-03-14 09:02:32.874984 | crc | changed: line added
2026-03-14 09:02:33.162233 | crc | changed: line added
2026-03-14 09:02:33.177996 | 
2026-03-14 09:02:33.178145 | TASK [prepare-crc-cloud : Increase dnsmasq cache size]
2026-03-14 09:02:33.493154 | crc | changed: line added
2026-03-14 09:02:33.509346 | 
2026-03-14 09:02:33.509482 | TASK [prepare-crc-cloud : Reload NetworkManager after adding cache and DNS server]
2026-03-14 09:02:34.471260 | crc | changed
2026-03-14 09:02:34.482887 | 
2026-03-14 09:02:34.482973 | TASK [prepare-crc-cloud : Ensure kube config does not exists]
2026-03-14 09:02:34.781488 | crc | ok
2026-03-14 09:02:34.802725 | 
2026-03-14 09:02:34.802914 | TASK [prepare-crc-cloud : Copy new generated KUBECONFIG]
2026-03-14 09:02:34.830326 | crc | skipping: Conditional result was False
2026-03-14 09:02:34.841418 | 
2026-03-14 09:02:34.841531 | TASK [prepare-crc-cloud : Delete all openshift-marketplace pods]
2026-03-14 09:02:35.329325 | crc | pod "certified-operators-j2vtm" deleted
2026-03-14 09:02:35.336585 | crc | pod "community-operators-8xw98" deleted
2026-03-14 09:02:35.345053 | crc | pod "marketplace-operator-79b997595-2hj4h" deleted
2026-03-14 09:02:35.360369 | crc | pod "redhat-marketplace-f4ztd" deleted
2026-03-14 09:02:35.363951 | crc | pod "redhat-operators-n2x6s" deleted
2026-03-14 09:02:37.396150 | crc | ok
2026-03-14 09:02:37.407100 | 
2026-03-14 09:02:37.407194 | TASK [prepare-crc-cloud : Create workaround for expired cert]
2026-03-14 09:02:37.489699 | crc | ok
2026-03-14 09:02:37.510642 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/cluster_cert.yaml
2026-03-14 09:02:37.555542 | 
2026-03-14 09:02:37.555660 | LOOP [prepare-crc-cloud : Re-run accept certificates in Pending state few times]
2026-03-14 09:02:38.081544 | crc | ok
2026-03-14 09:02:38.081850 | crc | changed: All items complete
2026-03-14 09:02:38.081891 | 
2026-03-14 09:02:48.507199 | crc | ok
2026-03-14 09:02:58.988690 | crc | ok
2026-03-14 09:02:59.058961 | 
2026-03-14 09:02:59.059141 | TASK [prepare-crc-cloud : Create service account for - auto-csr-approver]
2026-03-14 09:02:59.793741 | crc | ok
2026-03-14 09:02:59.803707 | 
2026-03-14 09:02:59.803817 | TASK [prepare-crc-cloud : Apply cluster cert approver]
2026-03-14 09:03:00.393122 | crc | serviceaccount/csr-approver-sa unchanged
2026-03-14 09:03:00.425122 | crc | clusterrole.rbac.authorization.k8s.io/csr-approver unchanged
2026-03-14 09:03:00.442664 | crc | clusterrolebinding.rbac.authorization.k8s.io/csr-approver-binding unchanged
2026-03-14 09:03:00.471506 | crc | cronjob.batch/auto-csr-approver unchanged
2026-03-14 09:03:00.857280 | crc | ok
2026-03-14 09:03:00.870690 | 
2026-03-14 09:03:00.870836 | TASK [prepare-crc-cloud : Pause machineconfigpool]
2026-03-14 09:03:00.914094 | crc | ok
2026-03-14 09:03:00.933516 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/pause_mcp.yaml
2026-03-14 09:03:00.979520 | 
2026-03-14 09:03:00.979651 | LOOP [prepare-crc-cloud : Pause the machineconfig configuration if mcp is not needed]
2026-03-14 09:03:01.541644 | crc | ok
2026-03-14 09:03:01.993047 | crc | ok
2026-03-14 09:03:02.007656 | 
2026-03-14 09:03:02.007794 | TASK [prepare-crc-cloud : Print current machineconfigpool]
2026-03-14 09:03:02.436803 | crc | NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
2026-03-14 09:03:02.444354 | crc | master   rendered-master-f83a149c7490c3b0bf850df839837383   False     True       True       1              0                   0                     1                      384d
2026-03-14 09:03:02.444391 | crc | worker   rendered-worker-594cc07c9feefb5e37f2530ac457d36c   True      False      False      0              0                   0                     0                      384d
2026-03-14 09:03:02.545500 | crc | ok
2026-03-14 09:03:02.561042 | 
2026-03-14 09:03:02.561190 | TASK [prepare-crc-cloud : Set low profile to etcd to avoid flakiness]
2026-03-14 09:03:02.588325 | crc | skipping: Conditional result was False
2026-03-14 09:03:02.605365 | 
2026-03-14 09:03:02.605537 | TASK [prepare-crc-cloud : Fix machineconfigpool degradation]
2026-03-14 09:03:02.642639 | crc | skipping: Conditional result was False
2026-03-14 09:03:02.660304 | 
2026-03-14 09:03:02.660455 | TASK [prepare-crc-cloud : Remove drain machineconfig for the node]
2026-03-14 09:03:02.697860 | crc | skipping: Conditional result was False
2026-03-14 09:03:02.716462 | 
2026-03-14 09:03:02.716625 | TASK [prepare-crc-cloud : Post machineconfigpool tasks]
2026-03-14 09:03:02.744627 | crc | skipping: Conditional result was False
2026-03-14 09:03:02.765699 | 
2026-03-14 09:03:02.765891 | TASK [prepare-crc-cloud : Enable monitoring and cluster-monitoring-operator]
2026-03-14 09:03:02.793856 | crc | skipping: Conditional result was False
2026-03-14 09:03:02.820304 | 
2026-03-14 09:03:02.820439 | TASK [Run crc-extracted preparation]
2026-03-14 09:03:02.845626 | crc | skipping: Conditional result was False
2026-03-14 09:03:02.865309 | 
2026-03-14 09:03:02.865429 | TASK [Fail when image build date does not exists]
2026-03-14 09:03:02.890699 | crc | skipping: Conditional result was False
2026-03-14 09:03:02.930192 | 
2026-03-14 09:03:02.930317 | PLAY [Set IOPS and Read/Write disk limitation]
2026-03-14 09:03:02.964937 | 
2026-03-14 09:03:02.965082 | TASK [Set limit for disk usage - IOPS and Read/Write]
2026-03-14 09:03:03.014888 | controller | ok
2026-03-14 09:03:03.160567 | crc | ok
2026-03-14 09:03:03.190259 | 
2026-03-14 09:03:03.190375 | TASK [disk-usage-limitation : Get block device major:minor]
2026-03-14 09:03:03.637495 | controller | 252:0
2026-03-14 09:03:03.514992 | crc | 252:0
2026-03-14 09:03:03.754746 | crc | ok
2026-03-14 09:03:03.768366 | controller | ok
2026-03-14 09:03:03.781695 | 
2026-03-14 09:03:03.781863 | LOOP [disk-usage-limitation : Ensure that some cgroup dirs exists]
2026-03-14 09:03:04.005596 | controller | ok: "/sys/fs/cgroup/init.scope"
2026-03-14 09:03:04.006074 | controller | ok: All items complete
2026-03-14 09:03:04.006144 | 
2026-03-14 09:03:04.113099 | crc | ok: "/sys/fs/cgroup/init.scope"
2026-03-14 09:03:04.189802 | controller | changed: "/sys/fs/cgroup/machine.slice"
2026-03-14 09:03:04.369570 | crc | ok: "/sys/fs/cgroup/machine.slice"
2026-03-14 09:03:04.388502 | controller | ok: "/sys/fs/cgroup/system.slice"
2026-03-14 09:03:04.597139 | controller | ok: "/sys/fs/cgroup/user.slice"
2026-03-14 09:03:04.612743 | crc | ok: "/sys/fs/cgroup/system.slice"
2026-03-14 09:03:04.873942 | crc | ok: "/sys/fs/cgroup/user.slice"
2026-03-14 09:03:04.885753 | 
2026-03-14 09:03:04.885877 | TASK [disk-usage-limitation : Ensure systemd configuration directory]
2026-03-14 09:03:05.153322 | controller | changed
2026-03-14 09:03:05.190892 | crc | ok
2026-03-14 09:03:05.196831 | 
2026-03-14 09:03:05.196894 | TASK [disk-usage-limitation : Enable IO limit via override configuration]
2026-03-14 09:03:05.799243 | controller | changed
2026-03-14 09:03:06.080986 | crc | changed
2026-03-14 09:03:06.094480 | 
2026-03-14 09:03:06.094662 | TASK [disk-usage-limitation : Flush handlers that will call systemd daemon-reload]
2026-03-14 09:03:06.117479 | 
2026-03-14 09:03:06.117557 | TASK [disk-usage-limitation : Flush handlers that will call systemd daemon-reload]
2026-03-14 09:03:06.133661 | 
2026-03-14 09:03:06.133754 | TASK [disk-usage-limitation : Reload systemd daemon after enabling IOAccounting]
2026-03-14 09:03:07.068586 | controller | ok
2026-03-14 09:03:08.023464 | crc | ok
2026-03-14 09:03:08.036209 | 
2026-03-14 09:03:08.036356 | TASK [disk-usage-limitation : Wait for system.slice cgroup to have io.max file]
2026-03-14 09:03:08.330371 | controller | ok
2026-03-14 09:03:08.350177 | crc | ok
2026-03-14 09:03:08.358445 | 
2026-03-14 09:03:08.358551 | LOOP [disk-usage-limitation : Set IOPS limit in the root cgroup]
2026-03-14 09:03:08.633427 | controller | ok
2026-03-14 09:03:08.633862 | controller | ok: All items complete
2026-03-14 09:03:08.633953 | 
2026-03-14 09:03:08.731329 | crc | ok
2026-03-14 09:03:08.828762 | controller | ok
2026-03-14 09:03:09.000595 | crc | ok
2026-03-14 09:03:09.044319 | controller | ok
2026-03-14 09:03:09.247644 | crc | ok
2026-03-14 09:03:09.256094 | controller | ok
2026-03-14 09:03:09.496465 | crc | ok
2026-03-14 09:03:09.516925 | 
2026-03-14 09:03:09.517177 | TASK [disk-usage-limitation : Verify the IOPS limit]
2026-03-14 09:03:09.827643 | controller | init
2026-03-14 09:03:09.831340 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:03:09.831381 | controller | machine
2026-03-14 09:03:09.831389 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:03:09.831395 | controller | system
2026-03-14 09:03:09.831410 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:03:09.831417 | controller | user
2026-03-14 09:03:09.831423 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:03:09.879184 | crc | init
2026-03-14 09:03:09.881203 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:03:09.881520 | crc | machine
2026-03-14 09:03:09.882759 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:03:09.882972 | crc | system
2026-03-14 09:03:09.885424 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:03:09.885755 | crc | user
2026-03-14 09:03:09.887029 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:03:10.126223 | controller | ok
2026-03-14 09:03:10.141502 | crc | ok
2026-03-14 09:03:10.156327 | 
2026-03-14 09:03:10.156465 | TASK [disk-usage-limitation : Check if kubepods.slice exists]
2026-03-14 09:03:10.363229 | controller | ok
2026-03-14 09:03:10.470717 | crc | ok
2026-03-14 09:03:10.484809 | 
2026-03-14 09:03:10.484958 | TASK [disk-usage-limitation : Set the limit]
2026-03-14 09:03:11.077731 | controller | skipping: Conditional result was False
2026-03-14 09:03:11.094566 | crc | ok
2026-03-14 09:03:11.108392 | 
2026-03-14 09:03:11.108532 | TASK [disk-usage-limitation : Verify the IOPS limit - kubepods slice]
2026-03-14 09:03:11.498585 | crc | kubepods
2026-03-14 09:03:11.500506 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:03:11.735111 | controller | skipping: Conditional result was False
2026-03-14 09:03:11.751070 | crc | ok
2026-03-14 09:03:11.763985 | 
2026-03-14 09:03:11.764193 | TASK [disk-usage-limitation : Print the current io.max value]
2026-03-14 09:03:11.842806 | controller | ok:
2026-03-14 09:03:11.843106 | controller | init
2026-03-14 09:03:11.843145 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:03:11.843173 | controller | machine
2026-03-14 09:03:11.843198 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:03:11.843222 | controller | system
2026-03-14 09:03:11.843245 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:03:11.843268 | controller | user
2026-03-14 09:03:11.843291 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:03:11.863556 | crc | ok:
2026-03-14 09:03:11.863626 | crc | init
2026-03-14 09:03:11.863659 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:03:11.863778 | crc | machine
2026-03-14 09:03:11.863838 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:03:11.863864 | crc | system
2026-03-14 09:03:11.863889 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:03:11.863913 | crc | user
2026-03-14 09:03:11.863936 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 kubepods
2026-03-14 09:03:11.863960 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000
2026-03-14 09:03:11.944230 | 
2026-03-14 09:03:11.944350 | PLAY RECAP
2026-03-14 09:03:11.944397 | controller | ok: 10 changed: 3 unreachable: 0 failed: 0 skipped: 3 rescued: 0 ignored: 0
2026-03-14 09:03:11.944422 | crc | ok: 112 changed: 37 unreachable: 0 failed: 0 skipped: 28 rescued: 0 ignored: 0
2026-03-14 09:03:11.944440 | 
2026-03-14 09:03:12.096244 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/prepare-crc.yaml@master]
2026-03-14 09:03:12.106275 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/podman.yaml@master]
2026-03-14 09:03:12.700752 | 
2026-03-14 09:03:12.700865 | PLAY [all,!compute]
2026-03-14 09:03:12.736428 | 
2026-03-14 09:03:12.736545 | TASK [Enable container-tools rhel-modules for c8/crc node]
2026-03-14 09:03:13.298547 | controller | skipping: Conditional result was False
2026-03-14 09:03:13.299678 | crc | skipping: Conditional result was False
2026-03-14 09:03:13.312930 | 
2026-03-14 09:03:13.313136 | TASK [Install podman]
2026-03-14 09:03:13.405647 | crc | skipping: Conditional result was False
2026-03-14 09:04:03.103517 | controller | changed
2026-03-14 09:04:03.116489 | 
2026-03-14 09:04:03.116642 | TASK [Set var name for quay login zuul secret]
2026-03-14 09:04:03.192835 | controller | skipping: Conditional result was False
2026-03-14 09:04:03.214535 | crc | skipping: Conditional result was False
2026-03-14 09:04:03.228386 | 
2026-03-14 09:04:03.228549 | TASK [Print the username]
2026-03-14 09:04:03.679163 | controller | openstack-k8s-operators+cirobot
2026-03-14 09:04:03.831344 | crc | skipping: Conditional result was False
2026-03-14 09:04:03.844516 | controller | ok: Runtime: 0:00:00.003625
2026-03-14 09:04:03.857552 | 
2026-03-14 09:04:03.857708 | TASK [Fail when user or password is not set]
2026-03-14 09:04:03.938890 | controller | skipping: Conditional result was False
2026-03-14 09:04:03.960433 | crc | skipping: Conditional result was False
2026-03-14 09:04:03.973930 | 
2026-03-14 09:04:03.974145 | TASK [Set vars for quay login]
2026-03-14 09:04:04.046421 | controller | Output suppressed because no_log was given
2026-03-14 09:04:04.058852 | 
2026-03-14 09:04:04.058999 | LOOP [Perform container registry login(s) with podman]
2026-03-14 09:04:04.893189 | controller | Output suppressed because no_log was given
2026-03-14 09:04:04.905578 | 
2026-03-14 09:04:04.905710 | LOOP [Perform container registry login(s) with buildah]
2026-03-14 09:04:04.982537 | controller | Output suppressed because no_log was given
2026-03-14 09:04:04.982972 | 
2026-03-14 09:04:04.999849 | 
2026-03-14 09:04:05.000233 | TASK [Set Insecure registry for content provider]
2026-03-14 09:04:05.102135 | crc | skipping: Conditional result was False
2026-03-14 09:04:05.449504 | controller | changed: Block inserted
2026-03-14 09:04:05.516588 | 
2026-03-14 09:04:05.516686 | PLAY RECAP
2026-03-14 09:04:05.516730 | controller | ok: 5 changed: 4 unreachable: 0 failed: 0 skipped: 4 rescued: 0 ignored: 0
2026-03-14 09:04:05.516757 | crc | ok: 0 changed: 0 unreachable: 0 failed: 0 skipped: 9 rescued: 0 ignored: 0
2026-03-14 09:04:05.516774 | 
2026-03-14 09:04:05.607080 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/podman.yaml@master]
2026-03-14 09:04:05.611589 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/data_plane_adoption/dpa_RH_subscription_registry_login.yaml@master]
2026-03-14 09:04:06.209400 | 
2026-03-14 09:04:06.209511 | PLAY [rh-subscription]
2026-03-14 09:04:06.209601 | 
2026-03-14 09:04:06.209636 | PLAY RECAP
2026-03-14 09:04:06.209665 | 
2026-03-14 09:04:06.287600 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/data_plane_adoption/dpa_RH_subscription_registry_login.yaml@master]
2026-03-14 09:04:06.297279 | PRE-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-customizations.yml@main]
2026-03-14 09:04:06.967065 | 
2026-03-14 09:04:06.967245 | PLAY [Tweak CRC node]
2026-03-14 09:04:06.994727 | 
2026-03-14 09:04:06.994818 | TASK [Load network parameters]
2026-03-14 09:04:07.593907 | crc | ok
2026-03-14 09:04:07.609185 | 
2026-03-14 09:04:07.609350 | TASK [Check which dnsmasq config we must edit]
2026-03-14 09:04:08.062922 | crc | ok
2026-03-14 09:04:08.072183 | 
2026-03-14 09:04:08.072359 | TASK [Ensure crc knows about its second NIC]
2026-03-14 09:04:09.199573 | crc | changed
2026-03-14 09:04:09.210293 | 
2026-03-14 09:04:09.210395 | TASK [Ensure crc does not get "public" DNS]
2026-03-14 09:04:10.272896 | crc | changed
2026-03-14 09:04:10.289568 | 
2026-03-14 09:04:10.289701 | TASK [Restart NetworkManager]
2026-03-14 09:04:14.118625 | crc | changed
2026-03-14 09:04:14.140342 | 
2026-03-14 09:04:14.140515 | LOOP [Configure dns forwarders]
2026-03-14 09:04:14.698182 | crc | ok
2026-03-14 09:04:14.947300 | crc | ok
2026-03-14 09:04:14.958449 | 
2026-03-14 09:04:14.958545 | TASK [Configure local DNS for CRC pod]
2026-03-14 09:04:17.390905 | crc | ok
2026-03-14 09:04:17.405770 | 
2026-03-14 09:04:17.405906 | TASK [Configure dnsmasq listen-address to listen on both br-ex and ci-private-network]
2026-03-14 09:04:17.749270 | crc | changed: line added
2026-03-14 09:04:17.758862 | 
2026-03-14 09:04:17.758970 | TASK [Restart native dnsmasq service]
2026-03-14 09:04:18.249600 | crc | changed
2026-03-14 09:04:18.268375 | 
2026-03-14 09:04:18.268506 | TASK [Stop dnsmasq]
2026-03-14 09:04:18.296239 | crc | skipping: Conditional result was False
2026-03-14 09:04:18.312405 | 
2026-03-14 09:04:18.312557 | TASK [Make sure that crc-dnsmasq is not running]
2026-03-14 09:04:18.339857 | crc | skipping: Conditional result was False
2026-03-14 09:04:18.358007 | 
2026-03-14 09:04:18.358200 | TASK [Start dnsmasq]
2026-03-14 09:04:18.385585 | crc | skipping: Conditional result was False
2026-03-14 09:04:18.404077 | 
2026-03-14 09:04:18.404257 | TASK [Wait for CRC to be ready]
2026-03-14 09:04:18.953568 | crc | WARNING: Using insecure TLS client config. Setting this option is not supported!
2026-03-14 09:04:19.109596 | crc |
2026-03-14 09:04:19.109658 | crc | Login successful.
2026-03-14 09:04:19.130289 | crc |
2026-03-14 09:04:19.130341 | crc | You have access to 65 projects, the list has been suppressed. You can list all projects with 'oc projects'
2026-03-14 09:04:19.130361 | crc |
2026-03-14 09:04:19.130369 | crc | Using project "default".
2026-03-14 09:04:19.464565 | crc | changed
2026-03-14 09:04:19.480735 | 
2026-03-14 09:04:19.480911 | TASK [Set insecure registry on crc node]
2026-03-14 09:04:19.533901 | crc | ok
2026-03-14 09:04:19.560502 | crc | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/ci/playbooks/tasks/set_crc_insecure_registry.yml
2026-03-14 09:04:19.577228 | 
2026-03-14 09:04:19.577385 | TASK [Patch the image.config.openshift.io resource to include insecure registry]
2026-03-14 09:04:20.050314 | crc | image.config.openshift.io/cluster patched
2026-03-14 09:04:20.162113 | crc | changed
2026-03-14 09:04:20.177271 | 
2026-03-14 09:04:20.177446 | TASK [Patch the image.config.openshift.io resource to allow registries]
2026-03-14 09:04:20.641018 | crc | image.config.openshift.io/cluster patched
2026-03-14 09:04:20.739924 | crc | changed
2026-03-14 09:04:20.755190 | 
2026-03-14 09:04:20.755376 | LOOP [Add additional allowed registries]
2026-03-14 09:04:20.812262 | 
2026-03-14 09:04:20.812554 | TASK [Ensure registries.conf.d exists]
2026-03-14 09:04:21.233865 | crc | ok
2026-03-14 09:04:21.249587 | 
2026-03-14 09:04:21.249768 | TASK [Set Insecure registry for content provider]
2026-03-14 09:04:21.705729 | crc | changed: File created
2026-03-14 09:04:21.721085 | 
2026-03-14 09:04:21.721225 | TASK [Set registry mirror override]
2026-03-14 09:04:21.758757 | crc | skipping: Conditional result was False
2026-03-14 09:04:21.775069 | 
2026-03-14 09:04:21.775256 | TASK [Restart crio]
2026-03-14 09:04:23.550165 | crc | changed
2026-03-14 09:04:23.607559 | 
2026-03-14 09:04:23.607696 | PLAY [Tweak Controller]
2026-03-14 09:04:23.633354 | 
2026-03-14 09:04:23.633466 | TASK [Install other packages]
2026-03-14 09:04:27.692478 | controller | changed
2026-03-14 09:04:27.705602 | 
2026-03-14 09:04:27.705742 | LOOP [Ensure we know compute host keys]
2026-03-14 09:04:28.135554 | controller | changed
2026-03-14 09:04:28.135995 | 
2026-03-14 09:04:28.409820 | controller | changed
2026-03-14 09:04:28.430696 | 
2026-03-14 09:04:28.430898 | TASK [Generate an ssh keypair]
2026-03-14 09:04:28.736693 | controller | Generating public/private ecdsa key pair.
2026-03-14 09:04:28.736937 | controller | Your identification has been saved in /home/zuul/.ssh/id_cifw
2026-03-14 09:04:28.736953 | controller | Your public key has been saved in /home/zuul/.ssh/id_cifw.pub
2026-03-14 09:04:28.736960 | controller | The key fingerprint is:
2026-03-14 09:04:28.736967 | controller | SHA256:i/MI9s5M2Flfugbzpa0QFMzkJLFqrh3NUZ+4TKsKfS4 zuul@np0005647129.novalocal
2026-03-14 09:04:28.736974 | controller | The key's randomart image is:
2026-03-14 09:04:28.736980 | controller | +---[ECDSA 256]---+
2026-03-14 09:04:28.736986 | controller | |      o=+        |
2026-03-14 09:04:28.736992 | controller | |       =o.       |
2026-03-14 09:04:28.736997 | controller | |      . +        |
2026-03-14 09:04:28.737003 | controller | |     . o o .     |
2026-03-14 09:04:28.737008 | controller | |    o . S o .    |
2026-03-14 09:04:28.737014 | controller | |   + = Bo* o.    |
2026-03-14 09:04:28.737019 | controller | |  . B X *+o+     |
2026-03-14 09:04:28.737025 | controller | |   =EX = .+..    |
2026-03-14 09:04:28.737031 | controller | |  . o+O ..o.     |
2026-03-14 09:04:28.737036 | controller | +----[SHA256]-----+
2026-03-14 09:04:28.989092 | controller | changed
2026-03-14 09:04:29.004103 | 
2026-03-14 09:04:29.004300 | TASK [Get public key]
2026-03-14 09:04:29.228821 | controller | ok
2026-03-14 09:04:29.242487 | 
2026-03-14 09:04:29.242624 | TASK [Register pub key as a fact]
2026-03-14 09:04:29.389965 | controller | ok
2026-03-14 09:04:29.397655 | 
2026-03-14 09:04:29.397735 | TASK [Load network parameters]
2026-03-14 09:04:29.601095 | controller | ok
2026-03-14 09:04:29.616388 | 
2026-03-14 09:04:29.616537 | TASK [Check if cloud-init is overriding NM settings]
2026-03-14 09:04:29.861726 | controller | ok
2026-03-14 09:04:29.875343 | 
2026-03-14 09:04:29.875478 | TASK [Remove cloud-init DNS override if present]
2026-03-14 09:04:29.902878 | controller | skipping: Conditional result was False
2026-03-14 09:04:29.916223 | 
2026-03-14 09:04:29.916361 | TASK [Reload the NetworkManager to pick the changes]
2026-03-14 09:04:29.943773 | controller | skipping: Conditional result was False
2026-03-14 09:04:29.956824 | 
2026-03-14 09:04:29.956955 | TASK [Get the default iface connection]
2026-03-14 09:04:30.238270 | controller | System eth0
2026-03-14 09:04:30.508210 | controller | changed
2026-03-14 09:04:30.522556 | 
2026-03-14 09:04:30.522736 | TASK [Prepend CRC DNS server in the controllers default Network Manager connection configuation]
2026-03-14 09:04:30.996878 | controller | Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/5)
2026-03-14 09:04:31.080607 | controller | changed
2026-03-14 09:04:31.127421 | 
2026-03-14 09:04:31.127538 | PLAY [Ensure persistent SSH access]
2026-03-14 09:04:31.168785 | 
2026-03-14 09:04:31.168917 | TASK [Inject key in ansible_user]
2026-03-14 09:04:31.647584 | controller | changed
2026-03-14 09:04:31.910378 | crc | changed
2026-03-14 09:04:31.928728 | 
2026-03-14 09:04:31.928971 | TASK [Inject key in root]
2026-03-14 09:04:32.258408 | controller | changed
2026-03-14 09:04:32.416598 | crc | changed
2026-03-14 09:04:32.494861 | 
2026-03-14 09:04:32.494970 | PLAY [Add cloud-admin user on Compute]
2026-03-14 09:04:32.496319 | 
2026-03-14 09:04:32.496368 | PLAY [Set some fancy hostname]
2026-03-14 09:04:32.512934 | 
2026-03-14 09:04:32.513038 | TASK [Inject hostname in configuration file]
2026-03-14 09:04:33.260772 | controller | changed
2026-03-14 09:04:33.310839 | 
2026-03-14 09:04:33.311059 | PLAY RECAP
2026-03-14 09:04:33.311165 | controller | ok: 12 changed: 8 unreachable: 0 failed: 0 skipped: 2 rescued: 0 ignored: 0
2026-03-14 09:04:33.311224 | crc | ok: 18 changed: 12 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0
2026-03-14 09:04:33.311264 | 
2026-03-14 09:04:33.445608 | PRE-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-customizations.yml@main]
2026-03-14 09:04:33.454694 | PRE-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/e2e-prepare.yml@main]
2026-03-14 09:04:34.051069 | 
2026-03-14 09:04:34.051179 | PLAY [Run ci/playbooks/e2e-prepare.yml]
2026-03-14 09:04:34.073275 | 
2026-03-14 09:04:34.073363 | TASK [Clone repos in the job workspace]
2026-03-14 09:04:34.103413 | controller | ok
2026-03-14 09:04:34.120854 | 
2026-03-14 09:04:34.120930 | TASK [prepare-workspace : Start zuul_console daemon.]
2026-03-14 09:04:34.638488 | controller | ok
2026-03-14 09:04:34.655239 | 
2026-03-14 09:04:34.655456 | TASK [prepare-workspace : Synchronize src repos to workspace directory.]
2026-03-14 09:04:45.732554 | controller | Output suppressed because no_log was given
2026-03-14 09:04:45.752065 | 
2026-03-14 09:04:45.752194 | TASK [Create zuul-output directory]
2026-03-14 09:04:46.146292 | controller | ok
2026-03-14 09:04:46.160095 | 
2026-03-14 09:04:46.160234 | TASK [Install required packages]
2026-03-14 09:04:47.211225 | controller | ok: Nothing to do
2026-03-14 09:04:47.218227 | 
2026-03-14 09:04:47.218301 | TASK [Output pip related things]
2026-03-14 09:04:47.816442 | controller | pip 21.3.1 from /usr/lib/python3.9/site-packages/pip (python 3.9)
2026-03-14 09:04:48.252106 | controller | changed
2026-03-14 09:04:48.265504 | 
2026-03-14 09:04:48.265653 | TASK [Install requirements]
2026-03-14 09:05:50.190165 | controller | changed
2026-03-14 09:05:50.204629 | 
2026-03-14 09:05:50.204706 | TASK [Add cronjob to trigger job stats]
2026-03-14 09:05:50.269648 | controller | skipping: Conditional result was False
2026-03-14 09:05:50.278990 | 
2026-03-14 09:05:50.279113 | LOOP [Construct project change list]
2026-03-14 09:05:50.352229 | controller | ok:
2026-03-14 09:05:50.352572 | controller | {
2026-03-14 09:05:50.352632 | controller |   "branch": "18.0-fr5",
2026-03-14 09:05:50.352676 | controller |   "change": "353",
2026-03-14 09:05:50.352716 | controller |   "change_message": "openstack-k8s-operators dependency bump branch: 18.0-fr5\n\nAutomated changes by [create-pull-request](https://github.com/peter-evans/create-pull-request) GitHub action",
2026-03-14 09:05:50.352764 | controller |   "change_url": "https://github.com/openstack-k8s-operators/watcher-operator/pull/353",
2026-03-14 09:05:50.352804 | controller |   "commit_id": "3fad4a9eb56718f26ce2ec186bb570f2695f01c3",
2026-03-14 09:05:50.352841 | controller |   "patchset": "3fad4a9eb56718f26ce2ec186bb570f2695f01c3",
2026-03-14 09:05:50.352878 | controller |   "project": {
2026-03-14 09:05:50.352920 | controller |     "canonical_hostname": "github.com",
2026-03-14 09:05:50.352973 | controller |     "canonical_name": "github.com/openstack-k8s-operators/watcher-operator",
2026-03-14 09:05:50.353073 | controller |     "name": "openstack-k8s-operators/watcher-operator",
2026-03-14 09:05:50.353114 | controller |     "short_name": "watcher-operator",
2026-03-14 09:05:50.353150 | controller |     "src_dir": "src/github.com/openstack-k8s-operators/watcher-operator"
2026-03-14 09:05:50.353188 | controller |   },
2026-03-14 09:05:50.353223 | controller |   "topic": null
2026-03-14 09:05:50.353261 | controller | }
2026-03-14 09:05:50.418383 | 
2026-03-14 09:05:50.418497 | PLAY [Run ci/playbooks/build_runner_image.yml]
2026-03-14 09:05:50.451521 | 
2026-03-14 09:05:50.451601 | TASK [Filter out host if needed]
2026-03-14 09:05:50.462267 | controller | skipping: end_host conditional evaluated to False, continuing execution for controller
2026-03-14 09:05:50.468752 | 
2026-03-14 09:05:50.468805 | TASK [Filter out host if needed]
2026-03-14 09:05:50.476573 | crc | skipping: end_host conditional evaluated to False, continuing execution for crc
2026-03-14 09:05:50.481685 | 
2026-03-14 09:05:50.481744 | TASK [Get git tag for image tagging]
2026-03-14 09:05:51.082712 | controller | skipping: Conditional result was False
2026-03-14 09:05:51.084146 | crc | skipping: Conditional result was False
2026-03-14 09:05:51.097288 | 
2026-03-14 09:05:51.097422 | TASK [Set openstack-runner image]
2026-03-14 09:05:51.139056 | controller | skipping: Conditional result was False
2026-03-14 09:05:51.151264 | crc | skipping: Conditional result was False
2026-03-14 09:05:51.166972 | 
2026-03-14 09:05:51.167174 | TASK [Build openstack-runner image]
2026-03-14 09:05:51.202721 | controller | skipping: Conditional result was False
2026-03-14 09:05:51.223924 | crc | skipping: Conditional result was False
2026-03-14 09:05:51.232614 | 
2026-03-14 09:05:51.232716 | TASK [Push openstack-runner image]
2026-03-14 09:05:51.266998 | controller | skipping: Conditional result was False
2026-03-14 09:05:51.278064 | crc | skipping: Conditional result was False
2026-03-14 09:05:51.286563 | 
2026-03-14 09:05:51.286648 | TASK [Make sure ci-framework directory exists]
2026-03-14 09:05:51.314291 | controller | skipping: Conditional result was False
2026-03-14 09:05:51.335304 | crc | skipping: Conditional result was False
2026-03-14 09:05:51.345053 | 
2026-03-14 09:05:51.345147 | TASK [Create EDPM ansible var file]
2026-03-14 09:05:51.376643 | controller | skipping: Conditional result was False
2026-03-14 09:05:51.397887 | crc | skipping: Conditional result was False
2026-03-14 09:05:51.456338 | 
2026-03-14 09:05:51.456454 | PLAY RECAP
2026-03-14 09:05:51.456500 | controller | ok: 7 changed: 3 unreachable: 0 failed: 0 skipped: 7 rescued: 0 ignored: 0
2026-03-14 09:05:51.456527 | crc | ok: 0 changed: 0 unreachable: 0 failed: 0 skipped: 6 rescued: 0 ignored: 0
2026-03-14 09:05:51.456546 | 
2026-03-14 09:05:51.558585 | PRE-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/e2e-prepare.yml@main]
2026-03-14 09:05:51.568197 | PRE-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/dump_zuul_data.yml@main]
2026-03-14 09:05:52.183367 | 
2026-03-14 09:05:52.183530 | PLAY [Run ci/playbooks/dump_zuul_data.yml]
2026-03-14 09:05:52.208145 | 
2026-03-14 09:05:52.208242 | TASK [Create zuul-output directory]
2026-03-14 09:05:52.757842 | controller | changed
2026-03-14 09:05:52.770480 | 
2026-03-14 09:05:52.770631 | TASK [Slurp Zuul inventory test]
2026-03-14 09:05:53.155419 | controller -> localhost | ok
2026-03-14 09:05:53.163080 | 
2026-03-14 09:05:53.163152 | TASK [Save zuul inventory]
2026-03-14 09:05:54.018927 | controller | changed
2026-03-14 09:05:54.032773 | 
2026-03-14 09:05:54.032961 | TASK [Save zuul vars without the change_message]
2026-03-14 09:05:55.202818 | controller | changed
2026-03-14 09:05:55.244283 | 
2026-03-14 09:05:55.244368 | PLAY RECAP
2026-03-14 09:05:55.244425 | controller | ok: 4 changed: 3 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0
2026-03-14 09:05:55.244454 | 
2026-03-14 09:05:55.337215 | PRE-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/dump_zuul_data.yml@main]
2026-03-14 09:05:55.341337 | PRE-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/kuttl-from-operator-deps.yaml@main]
2026-03-14 09:05:55.979057 | 
2026-03-14 09:05:55.979215 | PLAY [Run ci/playbooks/kuttl/deploy-deps.yml]
2026-03-14 09:05:56.003240 | 
2026-03-14 09:05:56.003335 | TASK [Run kuttl deploy-deps playbook]
2026-03-14 09:05:57.306807 | controller |
2026-03-14 09:05:58.480014 | controller | PLAY [Bootstrap playbook] ******************************************************
2026-03-14 09:05:58.480069 | controller |
2026-03-14 09:05:58.480075 | controller | TASK [Gathering Facts ] ********************************************************
2026-03-14 09:05:58.480080 | controller | Saturday 14 March 2026  09:05:57 +0000 (0:00:00.054)       0:00:00.054 ********
2026-03-14 09:05:58.480085 | controller | Saturday 14 March 2026  09:05:57 +0000 (0:00:00.052)       0:00:00.052 ********
2026-03-14 09:05:58.480094 | controller | ok: [controller]
2026-03-14 09:05:58.530283 | controller |
2026-03-14 09:05:58.530385 | controller | TASK [cifmw_setup : Set custom cifmw PATH reusable fact cifmw_path={{ ansible_user_dir }}/.crc/bin:{{ ansible_user_dir }}/.crc/bin/oc:{{ ansible_user_dir }}/bin:{{ ansible_env.PATH }}, cacheable=True] ***
2026-03-14 09:05:58.530400 | controller | Saturday 14 March 2026  09:05:58 +0000 (0:00:01.173)       0:00:01.228 ********
2026-03-14 09:05:58.530410 | controller | Saturday 14 March 2026  09:05:58 +0000 (0:00:01.173)       0:00:01.226 ********
2026-03-14 09:05:58.530426 | controller | ok: [controller]
2026-03-14 09:05:58.625229 | controller |
2026-03-14 09:05:58.625281 | controller | TASK [cifmw_setup : Get customized parameters ci_framework_params={{
2026-03-14 09:05:58.625302 | controller |   hostvars[inventory_hostname] |
2026-03-14 09:05:58.625308 | controller |   dict2items |
2026-03-14 09:05:58.625313 | controller |   selectattr("key", "match",
2026-03-14 09:05:58.625318 | controller |              "^(cifmw|pre|post)_(?!install_yamls|openshift_token|openshift_login|openshift_kubeconfig).*") |
2026-03-14 09:05:58.625323 | controller |   list | items2dict
2026-03-14 09:05:58.625327 | controller | }}] ***
2026-03-14 09:05:58.625332 | controller | Saturday 14 March 2026  09:05:58 +0000 (0:00:00.049)       0:00:01.277 ********
2026-03-14 09:05:58.625336 | controller | Saturday 14 March 2026  09:05:58 +0000 (0:00:00.049)       0:00:01.276 ********
2026-03-14 09:05:58.625347 | controller | ok: [controller]
2026-03-14 09:05:58.970437 | controller |
2026-03-14 09:05:58.970472 | controller | TASK [install_ca : Ensure target directory exists path={{ cifmw_install_ca_trust_dir }}, state=directory, mode=0755] ***
2026-03-14 09:05:58.970479 | controller | Saturday 14 March 2026  09:05:58 +0000 (0:00:00.095)       0:00:01.373 ********
2026-03-14 09:05:58.970484 | controller | Saturday 14 March 2026  09:05:58 +0000 (0:00:00.094)       0:00:01.371 ********
2026-03-14 09:05:58.970493 | controller | ok: [controller]
2026-03-14 09:05:58.992533 | controller |
2026-03-14 09:05:58.992585 | controller | TASK [install_ca : Install internal CA from url url={{ cifmw_install_ca_url }}, dest={{ cifmw_install_ca_trust_dir }}, validate_certs={{ cifmw_install_ca_url_validate_certs | default(omit) }}, mode=0644] ***
2026-03-14 09:05:58.992591 | controller | Saturday 14 March 2026  09:05:58 +0000 (0:00:00.345)       0:00:01.718 ********
2026-03-14 09:05:58.992598 | controller | Saturday 14 March 2026  09:05:58 +0000 (0:00:00.345)       0:00:01.717 ********
2026-03-14 09:05:58.992618 | controller | skipping: [controller]
2026-03-14 09:05:59.014984 | controller |
2026-03-14 09:05:59.015036 | controller | TASK [install_ca : Install custom CA bundle from inline dest={{ cifmw_install_ca_trust_dir }}/cifmw_inline_ca_bundle.crt, content={{ cifmw_install_ca_bundle_inline }}, mode=0644] ***
2026-03-14 09:05:59.015043 | controller | Saturday 14 March 2026  09:05:58 +0000 (0:00:00.022)       0:00:01.740 ********
2026-03-14 09:05:59.015049 | controller | Saturday 14 March 2026  09:05:58 +0000 (0:00:00.022)       0:00:01.739 ********
2026-03-14 09:05:59.015060 | controller | skipping: [controller]
2026-03-14 09:05:59.037767 | controller |
2026-03-14 09:05:59.037819 | controller | TASK [install_ca : Install custom CA bundle from file dest={{ cifmw_install_ca_trust_dir }}/{{ cifmw_install_ca_bundle_src | basename }}, src={{ cifmw_install_ca_bundle_src }}, mode=0644] ***
2026-03-14 09:05:59.037826 | controller | Saturday 14 March 2026  09:05:59 +0000 (0:00:00.022)       0:00:01.763 ********
2026-03-14 09:05:59.037832 | controller | Saturday 14 March 2026  09:05:59 +0000 (0:00:00.022)       0:00:01.761 ********
2026-03-14 09:05:59.037843 | controller | skipping: [controller]
2026-03-14 09:05:59.065171 | controller |
2026-03-14 09:05:59.065221 | controller | TASK [install_ca : Update ca bundle _raw_params=update-ca-trust] ***************
2026-03-14 09:05:59.065231 | controller | Saturday 14 March 2026  09:05:59 +0000 (0:00:00.022)       0:00:01.785 ********
2026-03-14 09:05:59.065246 | controller | Saturday 14 March 2026  09:05:59 +0000 (0:00:00.022)       0:00:01.784 ********
2026-03-14 09:05:59.065256 | controller | skipping: [controller]
2026-03-14 09:05:59.633409 | controller |
2026-03-14 09:05:59.633442 | controller | TASK [repo_setup : Ensure directories are present path={{ cifmw_repo_setup_basedir }}/{{ item }}, state=directory, mode=0755] ***
2026-03-14 09:05:59.633449 | controller | Saturday 14 March 2026  09:05:59 +0000 (0:00:00.027)       0:00:01.813 ********
2026-03-14 09:05:59.633454 | controller | Saturday 14 March 2026  09:05:59 +0000 (0:00:00.027)       0:00:01.811 ********
2026-03-14 09:05:59.633463 | controller | changed: [controller] => (item=tmp)
2026-03-14 09:06:00.591845 | controller | changed: [controller] => (item=artifacts/repositories)
2026-03-14 09:06:00.591881 | controller | changed: [controller] => (item=venv/repo_setup)
2026-03-14 09:06:00.591888 | controller |
2026-03-14 09:06:00.591893 | controller | TASK [repo_setup : Make sure git-core package is installed name=git-core, state=present] ***
2026-03-14 09:06:00.591898 | controller | Saturday 14 March 2026  09:05:59 +0000 (0:00:00.568)       0:00:02.381 ********
2026-03-14 09:06:00.591903 | controller | Saturday 14 March 2026  09:05:59 +0000 (0:00:00.568)       0:00:02.380 ********
2026-03-14 09:06:00.591912 | controller | ok: [controller]
2026-03-14 09:06:01.523076 | controller |
2026-03-14 09:06:01.523144 | controller | TASK [repo_setup : Get repo-setup repository accept_hostkey=True, dest={{ cifmw_repo_setup_basedir }}/tmp/repo-setup, repo={{ cifmw_repo_setup_src }}] ***
2026-03-14 09:06:01.523157 | controller | Saturday 14 March 2026  09:06:00 +0000 (0:00:00.958)       0:00:03.339 ********
2026-03-14 09:06:01.523167 | controller | Saturday 14 March 2026  09:06:00 +0000 (0:00:00.958)       0:00:03.338 ********
2026-03-14 09:06:01.523192 | controller | changed: [controller]
2026-03-14 09:06:09.957502 | controller |
2026-03-14 09:06:09.957588 | controller | TASK [repo_setup : Initialize python venv and install requirements virtualenv={{ cifmw_repo_setup_venv }}, requirements={{ cifmw_repo_setup_basedir }}/tmp/repo-setup/requirements.txt, virtualenv_command=python3 -m venv  --system-site-packages --upgrade-deps] ***
2026-03-14 09:06:09.957602 | controller | Saturday 14 March 2026  09:06:01 +0000 (0:00:00.930)       0:00:04.270 ********
2026-03-14 09:06:09.957613 | controller | Saturday 14 March 2026  09:06:01 +0000 (0:00:00.930)       0:00:04.269 ********
2026-03-14 09:06:09.957631 | controller | changed: [controller]
2026-03-14 09:06:10.818043 | controller |
2026-03-14 09:06:10.818081 | controller | TASK [repo_setup : Install repo-setup package chdir={{ cifmw_repo_setup_basedir }}/tmp/repo-setup, creates={{ cifmw_repo_setup_venv }}/bin/repo-setup, _raw_params={{ cifmw_repo_setup_venv }}/bin/python setup.py install] ***
2026-03-14 09:06:10.818096 | controller | Saturday 14 March 2026  09:06:09 +0000 (0:00:08.434)       0:00:12.705 ********
2026-03-14 09:06:10.818103 | controller | Saturday 14 March 2026  09:06:09 +0000 (0:00:08.434)       0:00:12.703 ********
2026-03-14 09:06:10.818117 | controller | changed: [controller]
2026-03-14 09:06:10.845326 | controller |
2026-03-14 09:06:10.845370 | controller | TASK [repo_setup : Set cifmw_repo_setup_dlrn_hash_tag from content provider cifmw_repo_setup_dlrn_hash_tag={{ content_provider_dlrn_md5_hash }}] ***
2026-03-14 09:06:10.845380 | controller | Saturday 14 March 2026  09:06:10 +0000 (0:00:00.860)       0:00:13.565 ********
2026-03-14 09:06:10.845388 | controller | Saturday 14 March 2026  09:06:10 +0000 (0:00:00.860)       0:00:13.564 ********
2026-03-14 09:06:10.845401 | controller | skipping: [controller]
2026-03-14 09:06:11.436923 | controller |
2026-03-14 09:06:11.436966 | controller | TASK [repo_setup : Run repo-setup _raw_params={{ cifmw_repo_setup_venv }}/bin/repo-setup {{ cifmw_repo_setup_promotion }} {{ cifmw_repo_setup_additional_repos }} -d {{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }} -b {{ cifmw_repo_setup_branch }} --rdo-mirror {{ cifmw_repo_setup_rdo_mirror }} {% if cifmw_repo_setup_dlrn_hash_tag | length > 0 %} --dlrn-hash-tag {{ cifmw_repo_setup_dlrn_hash_tag }} {% endif %} -o {{ cifmw_repo_setup_output }}] ***
2026-03-14 09:06:11.436976 | controller | Saturday 14 March 2026  09:06:10 +0000 (0:00:00.027)       0:00:13.593 ********
2026-03-14 09:06:11.436983 | controller | Saturday 14 March 2026  09:06:10 +0000 (0:00:00.027)       0:00:13.591 ********
2026-03-14 09:06:11.437002 | controller | changed: [controller]
2026-03-14 09:06:11.470888 | controller |
2026-03-14 09:06:11.470925 | controller | TASK [repo_setup : Get component repo url={{ cifmw_repo_setup_dlrn_uri }}/{{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }}-{{ cifmw_repo_setup_branch }}/component/{{ cifmw_repo_setup_component_name }}/{{ cifmw_repo_setup_component_promotion_tag }}/delorean.repo, dest={{ cifmw_repo_setup_output }}/{{ cifmw_repo_setup_component_name }}_{{ cifmw_repo_setup_component_promotion_tag }}_delorean.repo, mode=0644] ***
2026-03-14 09:06:11.470932 | controller | Saturday 14 March 2026  09:06:11 +0000 (0:00:00.591)       0:00:14.184 ********
2026-03-14 09:06:11.470937 | controller | Saturday 14 March 2026  09:06:11 +0000 (0:00:00.591)       0:00:14.183 ********
2026-03-14 09:06:11.470947 | controller | skipping: [controller]
2026-03-14 09:06:11.503338 | controller |
2026-03-14 09:06:11.503383 | controller | TASK [repo_setup : Rename component repo path={{ cifmw_repo_setup_output }}/{{ cifmw_repo_setup_component_name }}_{{ cifmw_repo_setup_component_promotion_tag }}_delorean.repo, regexp=delorean-component-{{ cifmw_repo_setup_component_name }}, replace={{ cifmw_repo_setup_component_name }}-{{ cifmw_repo_setup_component_promotion_tag }}] ***
2026-03-14 09:06:11.503393 | controller | Saturday 14 March 2026  09:06:11 +0000 (0:00:00.034)       0:00:14.218 ********
2026-03-14 09:06:11.503400 | controller | Saturday 14 March 2026  09:06:11 +0000 (0:00:00.034)       0:00:14.217 ********
2026-03-14 09:06:11.503412 | controller | skipping: [controller]
2026-03-14 09:06:11.533058 | controller |
2026-03-14 09:06:11.533098 | controller | TASK [repo_setup : Disable component repo in current-podified dlrn repo path={{ cifmw_repo_setup_output }}/delorean.repo, section=delorean-component-{{ cifmw_repo_setup_component_name }}, option=enabled, value=0, mode=0644] ***
2026-03-14 09:06:11.533107 | controller | Saturday 14 March 2026  09:06:11 +0000 (0:00:00.032)       0:00:14.251 ********
2026-03-14 09:06:11.533115 | controller | Saturday 14 March 2026  09:06:11 +0000 (0:00:00.032)       0:00:14.250 ********
2026-03-14 09:06:11.533127 | controller | skipping: [controller]
2026-03-14 09:06:11.970671 | controller |
2026-03-14 09:06:11.970702 | controller | TASK [repo_setup : Run repo-setup-get-hash _raw_params={{ cifmw_repo_setup_venv }}/bin/repo-setup-get-hash --dlrn-url {{ cifmw_repo_setup_dlrn_uri[:-1] }} --os-version {{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }} --release {{ cifmw_repo_setup_branch }} {% if cifmw_repo_setup_component_name | length > 0 -%} --component {{ cifmw_repo_setup_component_name }} --tag {{ cifmw_repo_setup_component_promotion_tag }} {% else -%} --tag {{cifmw_repo_setup_promotion }} {% endif -%} {% if (cifmw_repo_setup_dlrn_hash_tag | length > 0) and (cifmw_repo_setup_component_name | length <= 0) -%} --dlrn-hash-tag {{ cifmw_repo_setup_dlrn_hash_tag }} {% endif -%} --json] ***
2026-03-14 09:06:11.970709 | controller | Saturday 14 March 2026  09:06:11 +0000 (0:00:00.029)       0:00:14.281 ********
2026-03-14 09:06:11.970717 | controller | Saturday 14 March 2026  09:06:11 +0000 (0:00:00.029)       0:00:14.279 ********
2026-03-14 09:06:11.970726 | controller | changed: [controller]
2026-03-14 09:06:12.738880 | controller |
2026-03-14 09:06:12.738942 | controller | TASK [repo_setup : Dump full hash in delorean.repo.md5 file content={{ _repo_setup_json['full_hash'] }}
2026-03-14 09:06:12.738949 | controller | , dest={{ cifmw_repo_setup_basedir }}/artifacts/repositories/delorean.repo.md5, mode=0644] ***
2026-03-14 09:06:12.738961 | controller | Saturday 14 March 2026  09:06:11 +0000 (0:00:00.437)       0:00:14.718 ********
2026-03-14 09:06:12.738966 | controller | Saturday 14 March 2026  09:06:11 +0000 (0:00:00.437)       0:00:14.717 ********
2026-03-14 09:06:12.738976 | controller | changed: [controller]
2026-03-14 09:06:12.757552 | controller |
2026-03-14 09:06:12.757609 | controller | TASK [repo_setup : Dump current-podified hash url={{ cifmw_repo_setup_dlrn_uri }}/{{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }}-{{ cifmw_repo_setup_branch }}/current-podified/delorean.repo.md5, dest={{ cifmw_repo_setup_basedir }}/artifacts/repositories/delorean.repo.md5, mode=0644] ***
2026-03-14 09:06:12.757617 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.768)       0:00:15.487 ********
2026-03-14 09:06:12.757632 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.768)       0:00:15.485 ********
2026-03-14 09:06:12.757646 | controller | skipping: [controller]
2026-03-14 09:06:12.775025 | controller |
2026-03-14 09:06:12.775079 | controller | TASK [repo_setup : Slurp current podified hash src={{ cifmw_repo_setup_basedir }}/artifacts/repositories/delorean.repo.md5] ***
2026-03-14 09:06:12.775087 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.018)       0:00:15.505 ********
2026-03-14 09:06:12.775093 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.018)       0:00:15.504 ********
2026-03-14 09:06:12.775105 | controller | skipping: [controller]
2026-03-14 09:06:12.793087 | controller |
2026-03-14 09:06:12.793138 | controller | TASK [repo_setup : Update the value of full_hash _repo_setup_json={{ _repo_setup_json | combine({'full_hash': _hash}, recursive=true) }}] ***
2026-03-14 09:06:12.793146 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.017)       0:00:15.523 ********
2026-03-14 09:06:12.793152 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.017)       0:00:15.521 ********
2026-03-14 09:06:12.793164 | controller | skipping: [controller]
2026-03-14 09:06:12.823052 | controller |
2026-03-14 09:06:12.823108 | controller | TASK [repo_setup : Export hashes facts for further use cifmw_repo_setup_full_hash={{ _repo_setup_json['full_hash'] }}, cifmw_repo_setup_commit_hash={{ _repo_setup_json['commit_hash'] }}, cifmw_repo_setup_distro_hash={{ _repo_setup_json['distro_hash'] }}, cifmw_repo_setup_extended_hash={{ _repo_setup_json['extended_hash'] }}, cifmw_repo_setup_dlrn_api_url={{ _repo_setup_json['dlrn_api_url'] }}, cifmw_repo_setup_dlrn_url={{ _repo_setup_json['dlrn_url'] }}, cifmw_repo_setup_release={{ _repo_setup_json['release'] }}, cacheable=True] ***
2026-03-14 09:06:12.823119 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.017)       0:00:15.541 ********
2026-03-14 09:06:12.823126 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.018)       0:00:15.539 ********
2026-03-14 09:06:12.823140 | controller | ok: [controller]
2026-03-14 09:06:12.840988 | controller |
2026-03-14 09:06:12.841039 | controller | TASK [repo_setup : Create download directory path={{ cifmw_repo_setup_rhos_release_path }}, state=directory, mode=0755] ***
2026-03-14 09:06:12.841046 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.030)       0:00:15.571 ********
2026-03-14 09:06:12.841052 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.030)       0:00:15.569 ********
2026-03-14 09:06:12.841063 | controller | skipping: [controller]
2026-03-14 09:06:12.857850 | controller |
2026-03-14 09:06:12.857909 | controller | TASK [repo_setup : Print the URL to request msg={{ cifmw_repo_setup_rhos_release_rpm }}] ***
2026-03-14 09:06:12.857916 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.017)       0:00:15.588 ********
2026-03-14 09:06:12.857922 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.017)       0:00:15.587 ********
2026-03-14 09:06:12.857935 | controller | skipping: [controller]
2026-03-14 09:06:12.875872 | controller |
2026-03-14 09:06:12.875934 | controller | TASK [Download the RPM name=krb_request] ***************************************
2026-03-14 09:06:12.875943 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.016)       0:00:15.605 ********
2026-03-14 09:06:12.875949 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.016)       0:00:15.604 ********
2026-03-14 09:06:12.875964 | controller | skipping: [controller]
2026-03-14 09:06:12.892933 | controller |
2026-03-14 09:06:12.892989 | controller | TASK [repo_setup : Install RHOS Release tool name={{
2026-03-14 09:06:12.892998 | controller |   cifmw_repo_setup_rhos_release_rpm
2026-03-14 09:06:12.893004 | controller |   if cifmw_repo_setup_rhos_release_rpm is not url
2026-03-14 09:06:12.893008 | controller |   else cifmw_krb_request_out.path
2026-03-14 09:06:12.893013 | controller | }}, state=present, disable_gpg_check={{ cifmw_repo_setup_rhos_release_gpg_check | bool }}] ***
2026-03-14 09:06:12.893018 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.018)       0:00:15.623 ********
2026-03-14 09:06:12.893022 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.018)       0:00:15.622 ********
2026-03-14 09:06:12.893034 | controller | skipping: [controller]
2026-03-14 09:06:12.909136 | controller |
2026-03-14 09:06:12.909189 | controller | TASK [repo_setup : Get rhos-release tool version _raw_params=rhos-release --version] ***
2026-03-14 09:06:12.909204 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.017)       0:00:15.641 ********
2026-03-14 09:06:12.909210 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.017)       0:00:15.639 ********
2026-03-14 09:06:12.909222 | controller | skipping: [controller]
2026-03-14 09:06:12.926053 | controller |
2026-03-14 09:06:12.926112 | controller | TASK [repo_setup : Print rhos-release tool version msg={{ rr_version.stdout }}] ***
2026-03-14 09:06:12.926130 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.016)       0:00:15.657 ********
2026-03-14 09:06:12.926139 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.016)       0:00:15.656 ********
2026-03-14 09:06:12.926154 | controller | skipping: [controller]
2026-03-14 09:06:12.941799 | controller |
2026-03-14 09:06:12.941844 | controller | TASK [repo_setup : Generate repos using rhos-release {{ cifmw_repo_setup_rhos_release_args }} _raw_params=rhos-release {{ cifmw_repo_setup_rhos_release_args }} \
2026-03-14 09:06:12.941850 | controller |   -t {{ cifmw_repo_setup_output }}] ***
2026-03-14 09:06:12.941856 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.016)       0:00:15.674 ********
2026-03-14 09:06:12.941861 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.016)       0:00:15.672 ********
2026-03-14 09:06:12.941871 | controller | skipping: [controller]
2026-03-14 09:06:13.146635 | controller |
2026-03-14 09:06:13.146691 | controller | TASK [repo_setup : Check for /etc/ci/mirror_info.sh path=/etc/ci/mirror_info.sh] ***
2026-03-14 09:06:13.146699 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.015)       0:00:15.689 ********
2026-03-14 09:06:13.146704 | controller | Saturday 14 March 2026  09:06:12 +0000 (0:00:00.015)       0:00:15.688 ********
2026-03-14 09:06:13.146717 | controller | ok: [controller]
2026-03-14 09:06:13.358430 | controller |
2026-03-14 09:06:13.358464 | controller | TASK [repo_setup : Use RDO proxy mirrors chdir={{ cifmw_repo_setup_output }}, _raw_params=set -o pipefail
2026-03-14 09:06:13.358471 | controller | source /etc/ci/mirror_info.sh
2026-03-14 09:06:13.358477 | controller | sed -i -e "s|https://trunk.rdoproject.org|$NODEPOOL_RDO_PROXY|g" *.repo
2026-03-14 09:06:13.358481 | controller | ] ***
2026-03-14 09:06:13.358486 | controller | Saturday 14 March 2026  09:06:13 +0000 (0:00:00.204)       0:00:15.894 ********
2026-03-14 09:06:13.358491 | controller | Saturday 14 March 2026  09:06:13 +0000 (0:00:00.204)       0:00:15.893 ********
2026-03-14 09:06:13.358501 | controller | changed: [controller]
2026-03-14 09:06:13.589740 | controller |
2026-03-14 09:06:13.589787 | controller | TASK [repo_setup : Use RDO CentOS mirrors (remove CentOS 10 conditional when Nodepool mirrors exist) chdir={{ cifmw_repo_setup_output }}, _raw_params=set -o pipefail
2026-03-14 09:06:13.589794 | controller | source /etc/ci/mirror_info.sh
2026-03-14 09:06:13.589799 | controller | sed -i -e "s|http://mirror.stream.centos.org|$NODEPOOL_CENTOS_MIRROR|g" *.repo
2026-03-14 09:06:13.589804 | controller | ] ***
2026-03-14 09:06:13.589812 | controller | Saturday 14 March 2026  09:06:13 +0000 (0:00:00.211)       0:00:16.106 ********
2026-03-14 09:06:13.589817 | controller | Saturday 14 March 2026  09:06:13 +0000 (0:00:00.211)       0:00:16.105 ********
2026-03-14 09:06:13.589828 | controller | changed: [controller]
2026-03-14 09:06:14.189938 | controller |
2026-03-14 09:06:14.189973 | controller | TASK [repo_setup : Check for gating.repo file on content provider url=http://{{ content_provider_registry_ip }}:8766/gating.repo] ***
2026-03-14 09:06:14.189980 | controller | Saturday 14 March 2026  09:06:13 +0000 (0:00:00.230)       0:00:16.337 ********
2026-03-14 09:06:14.189985 | controller | Saturday 14 March 2026  09:06:13 +0000 (0:00:00.231)       0:00:16.336 ********
2026-03-14 09:06:14.189995 | controller | task path: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/repo_setup/tasks/populate_gating_repo.yml:2
2026-03-14 09:06:14.215523 | controller | fatal: [controller]: FAILED! =>
2026-03-14 09:06:14.215556 | controller |     changed: false
2026-03-14 09:06:14.215562 | controller |     elapsed: 0
2026-03-14 09:06:14.215568 | controller |     msg: 'Status code was -1 and not [200]: Request failed: <urlopen error [Errno 111]
2026-03-14 09:06:14.215573 | controller |       Connection refused>'
2026-03-14 09:06:14.215578 | controller |     redirected: false
2026-03-14 09:06:14.215582 | controller |     status: -1
2026-03-14 09:06:14.215589 | controller |     url: http://38.102.83.153:8766/gating.repo
2026-03-14 09:06:14.215600 | controller | ...ignoring
2026-03-14 09:06:14.215605 | controller |
2026-03-14 09:06:14.215610 | controller | TASK [repo_setup : Populate gating repo from content provider ip content=[gating-repo]
2026-03-14 09:06:14.215614 | controller | baseurl=http://{{ content_provider_registry_ip }}:8766/
2026-03-14 09:06:14.215619 | controller | enabled=1
2026-03-14 09:06:14.215625 | controller | gpgcheck=0
2026-03-14 09:06:14.215630 | controller | priority=1
2026-03-14 09:06:14.215634 | controller | , dest={{ cifmw_repo_setup_output }}/gating.repo, mode=0644] ***
2026-03-14 09:06:14.215639 | controller | Saturday 14 March 2026  09:06:14 +0000 (0:00:00.600)       0:00:16.938 ********
2026-03-14 09:06:14.215643 | controller | Saturday 14 March 2026  09:06:14 +0000 (0:00:00.600)       0:00:16.936 ********
2026-03-14 09:06:14.215652 | controller | skipping: [controller]
2026-03-14 09:06:14.240596 | controller |
2026-03-14 09:06:14.240627 | controller | TASK [repo_setup : Check for DLRN repo at the destination path={{ cifmw_repo_setup_output }}/delorean.repo] ***
2026-03-14 09:06:14.240634 | controller | Saturday 14 March 2026  09:06:14 +0000 (0:00:00.025)       0:00:16.963 ********
2026-03-14 09:06:14.240639 | controller | Saturday 14 March 2026  09:06:14 +0000 (0:00:00.025)       0:00:16.962 ********
2026-03-14 09:06:14.240649 | controller | skipping: [controller]
2026-03-14 09:06:14.268060 | controller |
2026-03-14 09:06:14.268094 | controller | TASK [repo_setup : Lower the priority of DLRN repos to allow installation from gating repo path={{ cifmw_repo_setup_output }}/delorean.repo, regexp=priority=1, replace=priority=20] ***
2026-03-14 09:06:14.268100 | controller | Saturday 14 March 2026  09:06:14 +0000 (0:00:00.025)       0:00:16.988 ********
2026-03-14 09:06:14.268106 | controller | Saturday 14 March 2026  09:06:14 +0000 (0:00:00.025)       0:00:16.987 ********
2026-03-14 09:06:14.268115 | controller | skipping: [controller]
2026-03-14 09:06:14.298122 | controller |
2026-03-14 09:06:14.298184 | controller | TASK [repo_setup : Check for DLRN component repo path={{ cifmw_repo_setup_output }}/{{ _comp_repo }}] ***
2026-03-14 09:06:14.298191 | controller | Saturday 14 March 2026  09:06:14 +0000 (0:00:00.027)       0:00:17.016 ********
2026-03-14 09:06:14.298196 | controller | Saturday 14 March 2026  09:06:14 +0000 (0:00:00.027)       0:00:17.014 ********
2026-03-14 09:06:14.298210 | controller | skipping: [controller]
2026-03-14 09:06:14.326652 | controller |
2026-03-14 09:06:14.326715 | controller | TASK [repo_setup : Lower the priority of component repos to allow installation from gating repo path={{ cifmw_repo_setup_output }}/{{ _comp_repo }}, regexp=priority=1, replace=priority=2] ***
2026-03-14 09:06:14.326724 | controller | Saturday 14 March 2026  09:06:14 +0000 (0:00:00.030)       0:00:17.046 ********
2026-03-14 09:06:14.326732 | controller | Saturday 14 March 2026  09:06:14 +0000 (0:00:00.030)       0:00:17.044 ********
2026-03-14 09:06:14.326750 | controller | skipping: [controller]
2026-03-14 09:06:14.620152 | controller |
2026-03-14 09:06:14.620205 | controller | TASK [repo_setup : Find existing repos from /etc/yum.repos.d directory paths=/etc/yum.repos.d/, patterns=*.repo, recurse=False] ***
2026-03-14 09:06:14.620212 | controller | Saturday 14 March 2026  09:06:14 +0000 (0:00:00.028)       0:00:17.074 ********
2026-03-14 09:06:14.620218 | controller | Saturday 14 March 2026  09:06:14 +0000 (0:00:00.028)       0:00:17.073 ********
2026-03-14 09:06:14.620230 | controller | ok: [controller]
2026-03-14 09:06:15.062593 | controller |
2026-03-14 09:06:15.062627 | controller | TASK [repo_setup : Remove existing repos from /etc/yum.repos.d directory path={{ item }}, state=absent] ***
2026-03-14 09:06:15.062634 | controller | Saturday 14 March 2026  09:06:14 +0000 (0:00:00.293)       0:00:17.368 ********
2026-03-14 09:06:15.062639 | controller | Saturday 14 March 2026  09:06:14 +0000 (0:00:00.293)       0:00:17.366 ********
2026-03-14 09:06:15.062648 | controller | changed: [controller] => (item=/etc/yum.repos.d/centos-addons.repo)
2026-03-14 09:06:15.459630 | controller | changed: [controller] => (item=/etc/yum.repos.d/centos.repo)
2026-03-14 09:06:15.459662 | controller |
2026-03-14 09:06:15.459668 | controller | TASK [repo_setup : Cleanup existing metadata _raw_params=dnf clean metadata] ***
2026-03-14 09:06:15.459673 | controller | Saturday 14 March 2026  09:06:15 +0000 (0:00:00.442)       0:00:17.810 ********
2026-03-14 09:06:15.459683 | controller | Saturday 14 March 2026  09:06:15 +0000 (0:00:00.442)       0:00:17.809 ********
2026-03-14 09:06:15.459692 | controller | changed: [controller]
2026-03-14 09:06:15.790811 | controller |
2026-03-14 09:06:15.790851 | controller | TASK [repo_setup : Copy generated repos to /etc/yum.repos.d directory mode=0755, remote_src=True, src={{ cifmw_repo_setup_output }}/, dest=/etc/yum.repos.d] ***
2026-03-14 09:06:15.790858 | controller | Saturday 14 March 2026  09:06:15 +0000 (0:00:00.397)       0:00:18.207 ********
2026-03-14 09:06:15.790863 | controller | Saturday 14 March 2026  09:06:15 +0000 (0:00:00.397)       0:00:18.206 ********
2026-03-14 09:06:15.790873 | controller | changed: [controller]
2026-03-14 09:06:15.828055 | controller |
2026-03-14 09:06:15.828088 | controller | TASK [ci_setup : Gather variables for each operating system _raw_params={{ item }}] ***
2026-03-14 09:06:15.828095 | controller | Saturday 14 March 2026  09:06:15 +0000 (0:00:00.331)       0:00:18.538 ********
2026-03-14 09:06:15.828100 | controller | Saturday 14 March 2026  09:06:15 +0000 (0:00:00.331)       0:00:18.537 ********
2026-03-14 09:06:15.828109 | controller | ok: [controller] => (item=/home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/ci_setup/vars/redhat.yml)
2026-03-14 09:06:15.849789 | controller |
2026-03-14 09:06:15.849820 | controller | TASK [ci_setup : List packages to install var=cifmw_ci_setup_packages] *********
2026-03-14 09:06:15.849826 | controller | Saturday 14 March 2026  09:06:15 +0000 (0:00:00.037)       0:00:18.576 ********
2026-03-14 09:06:15.849831 | controller | Saturday 14 March 2026  09:06:15 +0000 (0:00:00.037)       0:00:18.574 ********
2026-03-14 09:06:15.849840 | controller | ok: [controller] =>
2026-03-14 09:07:16.365753 | controller |     cifmw_ci_setup_packages:
2026-03-14 09:07:16.365789 | controller |     - bash-completion
2026-03-14 09:07:16.365796 | controller |     - ca-certificates
2026-03-14 09:07:16.365803 | controller |     - git-core
2026-03-14 09:07:16.365809 | controller |     - make
2026-03-14 09:07:16.365816 | controller |     - tar
2026-03-14 09:07:16.365821 | controller |     - tmux
2026-03-14 09:07:16.365827 | controller |     - python3-pip
2026-03-14 09:07:16.365833 | controller |
2026-03-14 09:07:16.365840 | controller | TASK [ci_setup : Install needed packages name={{ cifmw_ci_setup_packages }}, state=latest] ***
2026-03-14 09:07:16.365847 | controller | Saturday 14 March 2026  09:06:15 +0000 (0:00:00.021)       0:00:18.597 ********
2026-03-14 09:07:16.365853 | controller | Saturday 14 March 2026  09:06:15 +0000 (0:00:00.021)       0:00:18.596 ********
2026-03-14 09:07:16.365862 | controller | changed: [controller]
2026-03-14 09:07:16.564931 | controller |
2026-03-14 09:07:16.564963 | controller | TASK [ci_setup : Gather version of openshift client _raw_params=oc version --client -o yaml] ***
2026-03-14 09:07:16.564969 | controller | Saturday 14 March 2026  09:07:16 +0000 (0:01:00.516)       0:01:19.113 ********
2026-03-14 09:07:16.564974 | controller | Saturday 14 March 2026  09:07:16 +0000 (0:01:00.516)       0:01:19.112 ********
2026-03-14 09:07:16.564983 | controller | ok: [controller]
2026-03-14 09:07:16.771369 | controller |
2026-03-14 09:07:16.771418 | controller | TASK [ci_setup : Ensure openshift client install path is present path={{ cifmw_ci_setup_oc_install_path }}, state=directory, mode=0755] ***
2026-03-14 09:07:16.771428 | controller | Saturday 14 March 2026  09:07:16 +0000 (0:00:00.199)       0:01:19.313 ********
2026-03-14 09:07:16.771435 | controller | Saturday 14 March 2026  09:07:16 +0000 (0:00:00.199)       0:01:19.311 ********
2026-03-14 09:07:16.771449 | controller | changed: [controller]
2026-03-14 09:07:22.032824 | controller |
2026-03-14 09:07:22.032875 | controller | TASK [ci_setup : Install openshift client src={{ cifmw_ci_setup_openshift_client_download_uri }}/{{ cifmw_ci_setup_openshift_client_version }}/openshift-client-linux.tar.gz, dest={{ cifmw_ci_setup_oc_install_path }}, remote_src=True, mode=0755, creates={{ cifmw_ci_setup_oc_install_path }}/oc] ***
2026-03-14 09:07:22.032891 | controller | Saturday 14 March 2026  09:07:16 +0000 (0:00:00.206)       0:01:19.519 ********
2026-03-14 09:07:22.032900 | controller | Saturday 14 March 2026  09:07:16 +0000 (0:00:00.206)       0:01:19.518 ********
2026-03-14 09:07:22.032914 | controller | changed: [controller]
2026-03-14 09:07:22.061261 | controller |
2026-03-14 09:07:22.061354 | controller | TASK [ci_setup : Add the OC path to cifmw_path if needed cifmw_path={{ cifmw_ci_setup_oc_install_path }}:{{ ansible_env.PATH }}, cacheable=True] ***
2026-03-14 09:07:22.061366 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:05.261)       0:01:24.780 ********
2026-03-14 09:07:22.061373 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:05.261)       0:01:24.779 ********
2026-03-14 09:07:22.061393 | controller | skipping: [controller]
2026-03-14 09:07:22.404522 | controller |
2026-03-14 09:07:22.404556 | controller | TASK [ci_setup : Create completion file] ***************************************
2026-03-14 09:07:22.404562 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:00.028)       0:01:24.809 ********
2026-03-14 09:07:22.404567 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:00.028)       0:01:24.808 ********
2026-03-14 09:07:22.404577 | controller | changed: [controller]
2026-03-14 09:07:22.727989 | controller |
2026-03-14 09:07:22.728040 | controller | TASK [ci_setup : Source completion from within .bashrc create=True, mode=0644, path={{ ansible_user_dir }}/.bashrc, block=if [ -f ~/.oc_completion ]; then
2026-03-14 09:07:22.728054 | controller |   source ~/.oc_completion
2026-03-14 09:07:22.728066 | controller | fi] ***
2026-03-14 09:07:22.728077 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:00.343)       0:01:25.152 ********
2026-03-14 09:07:22.728087 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:00.343)       0:01:25.151 ********
2026-03-14 09:07:22.728105 | controller | changed: [controller]
2026-03-14 09:07:22.753118 | controller |
2026-03-14 09:07:22.753163 | controller | TASK [ci_setup : Check rhsm status _raw_params=subscription-manager status] ****
2026-03-14 09:07:22.753176 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:00.323)       0:01:25.475 ********
2026-03-14 09:07:22.753187 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:00.323)       0:01:25.474 ********
2026-03-14 09:07:22.753204 | controller | skipping: [controller]
2026-03-14 09:07:22.779278 | controller |
2026-03-14 09:07:22.779365 | controller | TASK [ci_setup : Gather the repos to be enabled _repos={{
2026-03-14 09:07:22.779378 | controller |   cifmw_ci_setup_rhel_rhsm_default_repos +
2026-03-14 09:07:22.779389 | controller |   (cifmw_ci_setup_rhel_rhsm_extra_repos | default([]))
2026-03-14 09:07:22.779399 | controller | }}] ***
2026-03-14 09:07:22.779409 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:00.025)       0:01:25.500 ********
2026-03-14 09:07:22.779420 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:00.025)       0:01:25.499 ********
2026-03-14 09:07:22.779437 | controller | skipping: [controller]
2026-03-14 09:07:22.804801 | controller |
2026-03-14 09:07:22.804845 | controller | TASK [ci_setup : Enabling the required repositories. name={{ item }}, state={{ rhsm_repo_state | default('enabled') }}] ***
2026-03-14 09:07:22.804858 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:00.025)       0:01:25.526 ********
2026-03-14 09:07:22.804876 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:00.025)       0:01:25.525 ********
2026-03-14 09:07:22.804892 | controller | skipping: [controller]
2026-03-14 09:07:22.831142 | controller |
2026-03-14 09:07:22.831179 | controller | TASK [ci_setup : Get current /etc/redhat-release _raw_params=cat /etc/redhat-release] ***
2026-03-14 09:07:22.831188 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:00.025)       0:01:25.552 ********
2026-03-14 09:07:22.831195 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:00.025)       0:01:25.551 ********
2026-03-14 09:07:22.831207 | controller | skipping: [controller]
2026-03-14 09:07:22.853815 | controller |
2026-03-14 09:07:22.853857 | controller | TASK [ci_setup : Print current /etc/redhat-release msg={{ _current_rh_release.stdout }}] ***
2026-03-14 09:07:22.853866 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:00.026)       0:01:25.578 ********
2026-03-14 09:07:22.853874 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:00.026)       0:01:25.577 ********
2026-03-14 09:07:22.853886 | controller | skipping: [controller]
2026-03-14 09:07:22.881097 | controller |
2026-03-14 09:07:22.881143 | controller | TASK [ci_setup : Ensure the repos are enabled in the system using yum name={{ item.name }}, baseurl={{ item.baseurl }}, description={{ item.description | default(item.name) }}, gpgcheck={{ item.gpgcheck | default(false) }}, enabled=True, state={{ yum_repo_state | default('present') }}] ***
2026-03-14 09:07:22.881152 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:00.022)       0:01:25.601 ********
2026-03-14 09:07:22.881160 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:00.022)       0:01:25.600 ********
2026-03-14 09:07:22.881172 | controller | skipping: [controller]
2026-03-14 09:07:23.995572 | controller |
2026-03-14 09:07:23.995623 | controller | TASK [ci_setup : Manage directories path={{ item }}, state={{ directory_state }}, mode=0755, owner={{ ansible_user_id }}, group={{ ansible_user_id }}] ***
2026-03-14 09:07:23.995648 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:00.027)       0:01:25.629 ********
2026-03-14 09:07:23.995659 | controller | Saturday 14 March 2026  09:07:22 +0000 (0:00:00.027)       0:01:25.627 ********
2026-03-14 09:07:23.995678 | controller | changed: [controller] => (item=/home/zuul/ci-framework-data/artifacts/manifests/openstack/cr)
2026-03-14 09:07:24.118354 | controller | changed: [controller] => (item=/home/zuul/ci-framework-data/logs)
2026-03-14 09:07:24.118409 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/tmp)
2026-03-14 09:07:24.118436 | controller | changed: [controller] => (item=/home/zuul/ci-framework-data/volumes)
2026-03-14 09:07:24.118452 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts/parameters)
2026-03-14 09:07:24.118467 | controller |
2026-03-14 09:07:24.118482 | controller | TASK [Prepare install_yamls make targets name=install_yamls, apply={'tags': ['bootstrap']}] ***
2026-03-14 09:07:24.118497 | controller | Saturday 14 March 2026  09:07:23 +0000 (0:00:01.114)       0:01:26.743 ********
2026-03-14 09:07:24.118513 | controller | Saturday 14 March 2026  09:07:23 +0000 (0:00:01.114)       0:01:26.742 ********
2026-03-14 09:07:24.118541 | controller |
2026-03-14 09:07:24.713724 | controller | TASK [install_yamls : Ensure directories exist path={{ item }}, state=directory, mode=0755] ***
2026-03-14 09:07:24.713776 | controller | Saturday 14 March 2026  09:07:24 +0000 (0:00:00.122)       0:01:26.866 ********
2026-03-14 09:07:24.713789 | controller | Saturday 14 March 2026  09:07:24 +0000 (0:00:00.122)       0:01:26.865 ********
2026-03-14 09:07:24.713808 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts)
2026-03-14 09:07:24.768894 | controller | changed: [controller] => (item=/home/zuul/ci-framework-data/artifacts/roles/install_yamls_makes/tasks)
2026-03-14 09:07:24.768932 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts/parameters)
2026-03-14 09:07:24.768941 | controller |
2026-03-14 09:07:24.768948 | controller | TASK [Create variables with local repos based on Zuul items name=install_yamls, tasks_from=zuul_set_operators_repo.yml] ***
2026-03-14 09:07:24.768956 | controller | Saturday 14 March 2026  09:07:24 +0000 (0:00:00.595)       0:01:27.461 ********
2026-03-14 09:07:24.768962 | controller | Saturday 14 March 2026  09:07:24 +0000 (0:00:00.595)       0:01:27.460 ********
2026-03-14 09:07:24.768975 | controller |
2026-03-14 09:07:24.815527 | controller | TASK [install_yamls : Set fact with local repos based on Zuul items cifmw_install_yamls_operators_repo={{ cifmw_install_yamls_operators_repo | default({}) | combine(_repo_operator_info | items2dict) }}] ***
2026-03-14 09:07:24.815558 | controller | Saturday 14 March 2026  09:07:24 +0000 (0:00:00.054)       0:01:27.516 ********
2026-03-14 09:07:24.815565 | controller | Saturday 14 March 2026  09:07:24 +0000 (0:00:00.054)       0:01:27.515 ********
2026-03-14 09:07:24.815576 | controller | ok: [controller] => (item={'branch': '18.0-fr5', 'change': '353', 'change_url': 'https://github.com/openstack-k8s-operators/watcher-operator/pull/353', 'commit_id': '3fad4a9eb56718f26ce2ec186bb570f2695f01c3', 'patchset': '3fad4a9eb56718f26ce2ec186bb570f2695f01c3', 'project': {'canonical_hostname': 'github.com', 'canonical_name': 'github.com/openstack-k8s-operators/watcher-operator', 'name': 'openstack-k8s-operators/watcher-operator', 'short_name': 'watcher-operator', 'src_dir': 'src/github.com/openstack-k8s-operators/watcher-operator'}, 'topic': None})
2026-03-14 09:07:24.880510 | controller |
2026-03-14 09:07:24.880576 | controller | TASK [install_yamls : Print helpful data for debugging msg=_repo_operator_name: {{ _repo_operator_name }}
2026-03-14 09:07:24.880598 | controller | _repo_operator_info: {{ _repo_operator_info }}
2026-03-14 09:07:24.880616 | controller | cifmw_install_yamls_operators_repo: {{ cifmw_install_yamls_operators_repo }}
2026-03-14 09:07:24.880632 | controller | ] ***
2026-03-14 09:07:24.880647 | controller | Saturday 14 March 2026  09:07:24 +0000 (0:00:00.046)       0:01:27.563 ********
2026-03-14 09:07:24.880659 | controller | Saturday 14 March 2026  09:07:24 +0000 (0:00:00.046)       0:01:27.562 ********
2026-03-14 09:07:24.880680 | controller | ok: [controller] => (item={'branch': '18.0-fr5', 'change': '353', 'change_url': 'https://github.com/openstack-k8s-operators/watcher-operator/pull/353', 'commit_id': '3fad4a9eb56718f26ce2ec186bb570f2695f01c3', 'patchset': '3fad4a9eb56718f26ce2ec186bb570f2695f01c3', 'project': {'canonical_hostname': 'github.com', 'canonical_name': 'github.com/openstack-k8s-operators/watcher-operator', 'name': 'openstack-k8s-operators/watcher-operator', 'short_name': 'watcher-operator', 'src_dir': 'src/github.com/openstack-k8s-operators/watcher-operator'}, 'topic': None}) =>
2026-03-14 09:07:24.931333 | controller |     msg: |
2026-03-14 09:07:24.931372 | controller |       _repo_operator_name: watcher
2026-03-14 09:07:24.931382 | controller |       _repo_operator_info: [{'key': 'WATCHER_REPO', 'value': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator'}, {'key': 'WATCHER_BRANCH', 'value': ''}]
2026-03-14 09:07:24.931390 | controller |       cifmw_install_yamls_operators_repo: {'WATCHER_REPO': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator', 'WATCHER_BRANCH': ''}
2026-03-14 09:07:24.931397 | controller |
2026-03-14 09:07:24.931405 | controller | TASK [Customize install_yamls devsetup vars if needed name=install_yamls, tasks_from=customize_devsetup_vars.yml] ***
2026-03-14 09:07:24.931412 | controller | Saturday 14 March 2026  09:07:24 +0000 (0:00:00.064)       0:01:27.627 ********
2026-03-14 09:07:24.931419 | controller | Saturday 14 March 2026  09:07:24 +0000 (0:00:00.064)       0:01:27.626 ********
2026-03-14 09:07:24.931432 | controller |
2026-03-14 09:07:24.951028 | controller | TASK [install_yamls : Update opm_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^opm_version:, line=opm_version: {{ cifmw_install_yamls_opm_version }}, state=present] ***
2026-03-14 09:07:24.951059 | controller | Saturday 14 March 2026  09:07:24 +0000 (0:00:00.051)       0:01:27.679 ********
2026-03-14 09:07:24.951067 | controller | Saturday 14 March 2026  09:07:24 +0000 (0:00:00.051)       0:01:27.677 ********
2026-03-14 09:07:24.951076 | controller | skipping: [controller]
2026-03-14 09:07:24.970522 | controller |
2026-03-14 09:07:24.970555 | controller | TASK [install_yamls : Update sdk_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^sdk_version:, line=sdk_version: {{ cifmw_install_yamls_sdk_version }}, state=present] ***
2026-03-14 09:07:24.970562 | controller | Saturday 14 March 2026  09:07:24 +0000 (0:00:00.020)       0:01:27.699 ********
2026-03-14 09:07:24.970567 | controller | Saturday 14 March 2026  09:07:24 +0000 (0:00:00.020)       0:01:27.697 ********
2026-03-14 09:07:24.970576 | controller | skipping: [controller]
2026-03-14 09:07:24.990634 | controller |
2026-03-14 09:07:24.990675 | controller | TASK [install_yamls : Update go_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^go_version:, line=go_version: {{ cifmw_install_yamls_go_version }}, state=present] ***
2026-03-14 09:07:24.990684 | controller | Saturday 14 March 2026  09:07:24 +0000 (0:00:00.019)       0:01:27.718 ********
2026-03-14 09:07:24.990690 | controller | Saturday 14 March 2026  09:07:24 +0000 (0:00:00.019)       0:01:27.717 ********
2026-03-14 09:07:24.990702 | controller | skipping: [controller]
2026-03-14 09:07:25.014761 | controller |
2026-03-14 09:07:25.014792 | controller | TASK [install_yamls : Update kustomize_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^kustomize_version:, line=kustomize_version: {{ cifmw_install_yamls_kustomize_version }}, state=present] ***
2026-03-14 09:07:25.014798 | controller | Saturday 14 March 2026  09:07:24 +0000 (0:00:00.020)       0:01:27.738 ********
2026-03-14 09:07:25.014810 | controller | Saturday 14 March 2026  09:07:24 +0000 (0:00:00.020)       0:01:27.737 ********
2026-03-14 09:07:25.014819 | controller | skipping: [controller]
2026-03-14 09:07:25.102445 | controller |
2026-03-14 09:07:25.102477 | controller | TASK [install_yamls : Compute the cifmw_install_yamls_vars final value _install_yamls_override_vars={{
2026-03-14 09:07:25.102483 | controller |   _install_yamls_override_vars | default({}) |
2026-03-14 09:07:25.102488 | controller |   combine(item, recursive=True)
2026-03-14 09:07:25.102493 | controller | }}] ***
2026-03-14 09:07:25.102505 | controller | Saturday 14 March 2026  09:07:25 +0000 (0:00:00.024)       0:01:27.762 ********
2026-03-14 09:07:25.102510 | controller | Saturday 14 March 2026  09:07:25 +0000 (0:00:00.024)       0:01:27.761 ********
2026-03-14 09:07:25.102518 | controller | ok: [controller] => (item={'BMO_SETUP': False})
2026-03-14 09:07:25.141352 | controller |
2026-03-14 09:07:25.141386 | controller | TASK [install_yamls : Set environment override cifmw_install_yamls_environment fact cifmw_install_yamls_environment={{
2026-03-14 09:07:25.141393 | controller |   _install_yamls_override_vars.keys() |
2026-03-14 09:07:25.141397 | controller |   map('upper') |
2026-03-14 09:07:25.141402 | controller |   zip(_install_yamls_override_vars.values()) |
2026-03-14 09:07:25.141406 | controller |   items2dict(key_name=0, value_name=1) |
2026-03-14 09:07:25.141411 | controller |   combine({
2026-03-14 09:07:25.141415 | controller |     'OUT': cifmw_install_yamls_manifests_dir,
2026-03-14 09:07:25.141420 | controller |     'OUTPUT_DIR': cifmw_install_yamls_edpm_dir,
2026-03-14 09:07:25.141424 | controller |     'CHECKOUT_FROM_OPENSTACK_REF': cifmw_install_yamls_checkout_openstack_ref,
2026-03-14 09:07:25.141430 | controller |     'OPENSTACK_K8S_BRANCH': (zuul is defined and not zuul.branch |regex_search('master|antelope|rhos')) | ternary(zuul.branch, 'main')
2026-03-14 09:07:25.141434 | controller |   }) |
2026-03-14 09:07:25.141439 | controller |   combine(install_yamls_operators_repos)
2026-03-14 09:07:25.141443 | controller | }}, cacheable=True] ***
2026-03-14 09:07:25.141448 | controller | Saturday 14 March 2026  09:07:25 +0000 (0:00:00.087)       0:01:27.850 ********
2026-03-14 09:07:25.141452 | controller | Saturday 14 March 2026  09:07:25 +0000 (0:00:00.087)       0:01:27.849 ********
2026-03-14 09:07:25.141461 | controller | ok: [controller]
2026-03-14 09:07:25.637044 | controller |
2026-03-14 09:07:25.637085 | controller | TASK [install_yamls : Get environment structure base_path={{ cifmw_install_yamls_repo }}] ***
2026-03-14 09:07:25.637094 | controller | Saturday 14 March 2026  09:07:25 +0000 (0:00:00.038)       0:01:27.889 ********
2026-03-14 09:07:25.637102 | controller | Saturday 14 March 2026  09:07:25 +0000 (0:00:00.038)       0:01:27.888 ********
2026-03-14 09:07:25.637114 | controller | ok: [controller]
2026-03-14 09:07:25.847139 | controller |
2026-03-14 09:07:25.847236 | controller | TASK [install_yamls : Ensure Output directory exists path={{ cifmw_install_yamls_out_dir }}, state=directory, mode=0755] ***
2026-03-14 09:07:25.847246 | controller | Saturday 14 March 2026  09:07:25 +0000 (0:00:00.495)       0:01:28.384 ********
2026-03-14 09:07:25.847253 | controller | Saturday 14 March 2026  09:07:25 +0000 (0:00:00.495)       0:01:28.383 ********
2026-03-14 09:07:25.847266 | controller | ok: [controller]
2026-03-14 09:07:25.890727 | controller |
2026-03-14 09:07:25.890766 | controller | TASK [install_yamls : Ensure user cifmw_install_yamls_vars contains existing Makefile variables that=_cifmw_install_yamls_unmatched_vars | length == 0, msg=cifmw_install_yamls_vars contains a variable that is not defined in install_yamls Makefile nor cifmw_install_yamls_whitelisted_vars: {{ _cifmw_install_yamls_unmatched_vars | join(', ')}}, quiet=True] ***
2026-03-14 09:07:25.890775 | controller | Saturday 14 March 2026  09:07:25 +0000 (0:00:00.210)       0:01:28.595 ********
2026-03-14 09:07:25.890782 | controller | Saturday 14 March 2026  09:07:25 +0000 (0:00:00.210)       0:01:28.593 ********
2026-03-14 09:07:25.890794 | controller | ok: [controller]
2026-03-14 09:07:26.485891 | controller |
2026-03-14 09:07:26.485934 | controller | TASK [install_yamls : Generate /home/zuul/ci-framework-data/artifacts/install_yamls.sh dest={{ cifmw_install_yamls_out_dir }}/{{ cifmw_install_yamls_envfile }}, content={% for k,v in cifmw_install_yamls_environment.items() %}
2026-03-14 09:07:26.485943 | controller | export {{ k }}={{ v }}
2026-03-14 09:07:26.485950 | controller | {% endfor %}, mode=0644] ***
2026-03-14 09:07:26.485956 | controller | Saturday 14 March 2026  09:07:25 +0000 (0:00:00.043)       0:01:28.638 ********
2026-03-14 09:07:26.485963 | controller | Saturday 14 March 2026  09:07:25 +0000 (0:00:00.043)       0:01:28.637 ********
2026-03-14 09:07:26.485976 | controller | changed: [controller]
2026-03-14 09:07:26.517132 | controller |
2026-03-14 09:07:26.517167 | controller | TASK [install_yamls : Set install_yamls default values cifmw_install_yamls_defaults={{ get_makefiles_env_output.makefiles_values  | combine(cifmw_install_yamls_environment) }}, cacheable=True] ***
2026-03-14 09:07:26.517182 | controller | Saturday 14 March 2026  09:07:26 +0000 (0:00:00.594)       0:01:29.233 ********
2026-03-14 09:07:26.517191 | controller | Saturday 14 March 2026  09:07:26 +0000 (0:00:00.594)       0:01:29.232 ********
2026-03-14 09:07:26.517203 | controller | ok: [controller]
2026-03-14 09:07:26.541047 | controller |
2026-03-14 09:07:26.541081 | controller | TASK [install_yamls : Show the env structure var=cifmw_install_yamls_environment] ***
2026-03-14 09:07:26.541091 | controller | Saturday 14 March 2026  09:07:26 +0000 (0:00:00.031)       0:01:29.265 ********
2026-03-14 09:07:26.541098 | controller | Saturday 14 March 2026  09:07:26 +0000 (0:00:00.031)       0:01:29.263 ********
2026-03-14 09:07:26.541109 | controller | ok: [controller] =>
2026-03-14 09:07:26.568865 | controller |     cifmw_install_yamls_environment:
2026-03-14 09:07:26.568904 | controller |       BMO_SETUP: false
2026-03-14 09:07:26.568912 | controller |       CHECKOUT_FROM_OPENSTACK_REF: 'true'
2026-03-14 09:07:26.568924 | controller |       OPENSTACK_K8S_BRANCH: 18.0-fr5
2026-03-14 09:07:26.568931 | controller |       OUT: /home/zuul/ci-framework-data/artifacts/manifests
2026-03-14 09:07:26.568938 | controller |       OUTPUT_DIR: /home/zuul/ci-framework-data/artifacts/edpm
2026-03-14 09:07:26.568945 | controller |       WATCHER_BRANCH: ''
2026-03-14 09:07:26.568951 | controller |       WATCHER_REPO: /home/zuul/src/github.com/openstack-k8s-operators/watcher-operator
2026-03-14 09:07:26.568958 | controller |
2026-03-14 09:07:26.568964 | controller | TASK [install_yamls : Show the env structure defaults var=cifmw_install_yamls_defaults] ***
2026-03-14 09:07:26.568971 | controller | Saturday 14 March 2026  09:07:26 +0000 (0:00:00.023)       0:01:29.289 ********
2026-03-14 09:07:26.568977 | controller | Saturday 14 March 2026  09:07:26 +0000 (0:00:00.023)       0:01:29.287 ********
2026-03-14 09:07:26.568989 | controller | ok: [controller] =>
2026-03-14 09:07:26.568996 | controller |     cifmw_install_yamls_defaults:
2026-03-14 09:07:26.569002 | controller |       ADOPTED_EXTERNAL_NETWORK: 172.21.1.0/24
2026-03-14 09:07:26.569009 | controller |       ADOPTED_INTERNALAPI_NETWORK: 172.17.1.0/24
2026-03-14 09:07:26.569015 | controller |       ADOPTED_STORAGEMGMT_NETWORK: 172.20.1.0/24
2026-03-14 09:07:26.569021 | controller |       ADOPTED_STORAGE_NETWORK: 172.18.1.0/24
2026-03-14 09:07:26.569027 | controller |       ADOPTED_TENANT_NETWORK: 172.9.1.0/24
2026-03-14 09:07:26.569033 | controller |       BARBICAN: config/samples/barbican_v1beta1_barbican.yaml
2026-03-14 09:07:26.569039 | controller |       BARBICAN_BRANCH: 18.0-fr5
2026-03-14 09:07:26.569045 | controller |       BARBICAN_COMMIT_HASH: ''
2026-03-14 09:07:26.569051 | controller |       BARBICAN_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/barbican-operator/config/samples/barbican_v1beta1_barbican.yaml
2026-03-14 09:07:26.569058 | controller |       BARBICAN_DEPL_IMG: unused
2026-03-14 09:07:26.569064 | controller |       BARBICAN_IMG: quay.io/openstack-k8s-operators/barbican-operator-index:latest
2026-03-14 09:07:26.569070 | controller |       BARBICAN_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/barbican-operator/kuttl-test.yaml
2026-03-14 09:07:26.569077 | controller |       BARBICAN_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/barbican-operator/test/kuttl/tests
2026-03-14 09:07:26.569083 | controller |       BARBICAN_KUTTL_NAMESPACE: barbican-kuttl-tests
2026-03-14 09:07:26.569089 | controller |       BARBICAN_REPO: https://github.com/openstack-k8s-operators/barbican-operator.git
2026-03-14 09:07:26.569095 | controller |       BARBICAN_SERVICE_ENABLED: 'true'
2026-03-14 09:07:26.569101 | controller |       BARBICAN_SIMPLE_CRYPTO_ENCRYPTION_KEY: sEFmdFjDUqRM2VemYslV5yGNWjokioJXsg8Nrlc3drU=
2026-03-14 09:07:26.569107 | controller |       BAREMETAL_BRANCH: 18.0-fr5
2026-03-14 09:07:26.569113 | controller |       BAREMETAL_COMMIT_HASH: ''
2026-03-14 09:07:26.569119 | controller |       BAREMETAL_IMG: quay.io/openstack-k8s-operators/openstack-baremetal-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.569125 | controller |       BAREMETAL_OS_CONTAINER_IMG: ''
2026-03-14 09:07:26.569131 | controller |       BAREMETAL_OS_IMG: ''
2026-03-14 09:07:26.569137 | controller |       BAREMETAL_OS_IMG_TYPE: ''
2026-03-14 09:07:26.569143 | controller |       BAREMETAL_REPO: https://github.com/openstack-k8s-operators/openstack-baremetal-operator.git
2026-03-14 09:07:26.569149 | controller |       BAREMETAL_TIMEOUT: 20m
2026-03-14 09:07:26.569155 | controller |       BASH_IMG: quay.io/openstack-k8s-operators/bash:latest
2026-03-14 09:07:26.569168 | controller |       BGP_ASN: '64999'
2026-03-14 09:07:26.569174 | controller |       BGP_LEAF_1: 100.65.4.1
2026-03-14 09:07:26.569180 | controller |       BGP_LEAF_2: 100.64.4.1
2026-03-14 09:07:26.569210 | controller |       BGP_OVN_ROUTING: 'false'
2026-03-14 09:07:26.569216 | controller |       BGP_PEER_ASN: '64999'
2026-03-14 09:07:26.569222 | controller |       BGP_SOURCE_IP: 172.30.4.2
2026-03-14 09:07:26.569228 | controller |       BGP_SOURCE_IP6: f00d:f00d:f00d:f00d:f00d:f00d:f00d:42
2026-03-14 09:07:26.569234 | controller |       BMAAS_BRIDGE_IPV4_PREFIX: 172.20.1.2/24
2026-03-14 09:07:26.569240 | controller |       BMAAS_BRIDGE_IPV6_PREFIX: fd00:bbbb::2/64
2026-03-14 09:07:26.569246 | controller |       BMAAS_INSTANCE_DISK_SIZE: '20'
2026-03-14 09:07:26.569252 | controller |       BMAAS_INSTANCE_MEMORY: '4096'
2026-03-14 09:07:26.569258 | controller |       BMAAS_INSTANCE_NAME_PREFIX: crc-bmaas
2026-03-14 09:07:26.569264 | controller |       BMAAS_INSTANCE_NET_MODEL: virtio
2026-03-14 09:07:26.569271 | controller |       BMAAS_INSTANCE_OS_VARIANT: centos-stream9
2026-03-14 09:07:26.569277 | controller |       BMAAS_INSTANCE_VCPUS: '2'
2026-03-14 09:07:26.569283 | controller |       BMAAS_INSTANCE_VIRT_TYPE: kvm
2026-03-14 09:07:26.569310 | controller |       BMAAS_IPV4: 'true'
2026-03-14 09:07:26.569317 | controller |       BMAAS_IPV6: 'false'
2026-03-14 09:07:26.569417 | controller |       BMAAS_LIBVIRT_USER: sushyemu
2026-03-14 09:07:26.569427 | controller |       BMAAS_METALLB_ADDRESS_POOL: 172.20.1.64/26
2026-03-14 09:07:26.569435 | controller |       BMAAS_METALLB_POOL_NAME: baremetal
2026-03-14 09:07:26.569443 | controller |       BMAAS_NETWORK_IPV4_PREFIX: 172.20.1.1/24
2026-03-14 09:07:26.569451 | controller |       BMAAS_NETWORK_IPV6_PREFIX: fd00:bbbb::1/64
2026-03-14 09:07:26.569459 | controller |       BMAAS_NETWORK_NAME: crc-bmaas
2026-03-14 09:07:26.569465 | controller |       BMAAS_NODE_COUNT: '1'
2026-03-14 09:07:26.569471 | controller |       BMAAS_OCP_INSTANCE_NAME: crc
2026-03-14 09:07:26.569477 | controller |       BMAAS_REDFISH_PASSWORD: password
2026-03-14 09:07:26.569483 | controller |       BMAAS_REDFISH_USERNAME: admin
2026-03-14 09:07:26.569488 | controller |       BMAAS_ROUTE_LIBVIRT_NETWORKS: crc-bmaas,crc,default
2026-03-14 09:07:26.569495 | controller |       BMAAS_SUSHY_EMULATOR_DRIVER: libvirt
2026-03-14 09:07:26.569501 | controller |       BMAAS_SUSHY_EMULATOR_IMAGE: quay.io/metal3-io/sushy-tools:latest
2026-03-14 09:07:26.569507 | controller |       BMAAS_SUSHY_EMULATOR_NAMESPACE: sushy-emulator
2026-03-14 09:07:26.569512 | controller |       BMAAS_SUSHY_EMULATOR_OS_CLIENT_CONFIG_FILE: /etc/openstack/clouds.yaml
2026-03-14 09:07:26.569518 | controller |       BMAAS_SUSHY_EMULATOR_OS_CLOUD: openstack
2026-03-14 09:07:26.569524 | controller |       BMH_NAMESPACE: openstack
2026-03-14 09:07:26.569530 | controller |       BMO_BRANCH: release-0.9
2026-03-14 09:07:26.569536 | controller |       BMO_CLEANUP: 'true'
2026-03-14 09:07:26.569542 | controller |       BMO_COMMIT_HASH: ''
2026-03-14 09:07:26.569548 | controller |       BMO_IPA_BASEURI: https://tarballs.opendev.org/openstack/ironic-python-agent/dib
2026-03-14 09:07:26.569554 | controller |       BMO_IPA_BRANCH: stable/2024.1
2026-03-14 09:07:26.569560 | controller |       BMO_IPA_FLAVOR: centos9
2026-03-14 09:07:26.569566 | controller |       BMO_IPA_INSECURE: 'false'
2026-03-14 09:07:26.569572 | controller |       BMO_IRONIC_HOST: 192.168.122.10
2026-03-14 09:07:26.569578 | controller |       BMO_PROVISIONING_INTERFACE: ''
2026-03-14 09:07:26.569584 | controller |       BMO_REPO: https://github.com/metal3-io/baremetal-operator
2026-03-14 09:07:26.569590 | controller |       BMO_SETUP: false
2026-03-14 09:07:26.569596 | controller |       BMO_SETUP_ROUTE_REPLACE: 'true'
2026-03-14 09:07:26.569602 | controller |       BM_CTLPLANE_INTERFACE: enp1s0
2026-03-14 09:07:26.569608 | controller |       BM_INSTANCE_MEMORY: '8192'
2026-03-14 09:07:26.569613 | controller |       BM_INSTANCE_NAME_PREFIX: edpm-compute-baremetal
2026-03-14 09:07:26.569619 | controller |       BM_INSTANCE_NAME_SUFFIX: '0'
2026-03-14 09:07:26.569625 | controller |       BM_NETWORK_NAME: default
2026-03-14 09:07:26.569631 | controller |       BM_NODE_COUNT: '1'
2026-03-14 09:07:26.569637 | controller |       BM_ROOT_PASSWORD: ''
2026-03-14 09:07:26.569643 | controller |       BM_ROOT_PASSWORD_SECRET: ''
2026-03-14 09:07:26.569649 | controller |       CEILOMETER_CENTRAL_DEPL_IMG: unused
2026-03-14 09:07:26.569655 | controller |       CEILOMETER_NOTIFICATION_DEPL_IMG: unused
2026-03-14 09:07:26.569661 | controller |       CEPH_BRANCH: release-1.15
2026-03-14 09:07:26.569667 | controller |       CEPH_CLIENT: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/toolbox.yaml
2026-03-14 09:07:26.569684 | controller |       CEPH_COMMON: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/common.yaml
2026-03-14 09:07:26.569690 | controller |       CEPH_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/cluster-test.yaml
2026-03-14 09:07:26.569696 | controller |       CEPH_CRDS: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/crds.yaml
2026-03-14 09:07:26.569702 | controller |       CEPH_IMG: quay.io/ceph/demo:latest-squid
2026-03-14 09:07:26.569708 | controller |       CEPH_OP: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/operator-openshift.yaml
2026-03-14 09:07:26.569714 | controller |       CEPH_REPO: https://github.com/rook/rook.git
2026-03-14 09:07:26.569720 | controller |       CERTMANAGER_TIMEOUT: 300s
2026-03-14 09:07:26.569726 | controller |       CHECKOUT_FROM_OPENSTACK_REF: 'true'
2026-03-14 09:07:26.569731 | controller |       CINDER: config/samples/cinder_v1beta1_cinder.yaml
2026-03-14 09:07:26.569737 | controller |       CINDERAPI_DEPL_IMG: unused
2026-03-14 09:07:26.569743 | controller |       CINDERBKP_DEPL_IMG: unused
2026-03-14 09:07:26.569749 | controller |       CINDERSCH_DEPL_IMG: unused
2026-03-14 09:07:26.569755 | controller |       CINDERVOL_DEPL_IMG: unused
2026-03-14 09:07:26.569761 | controller |       CINDER_BRANCH: 18.0-fr5
2026-03-14 09:07:26.569767 | controller |       CINDER_COMMIT_HASH: ''
2026-03-14 09:07:26.569773 | controller |       CINDER_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/cinder-operator/config/samples/cinder_v1beta1_cinder.yaml
2026-03-14 09:07:26.569779 | controller |       CINDER_IMG: quay.io/openstack-k8s-operators/cinder-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.569785 | controller |       CINDER_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/cinder-operator/kuttl-test.yaml
2026-03-14 09:07:26.569791 | controller |       CINDER_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/cinder-operator/test/kuttl/tests
2026-03-14 09:07:26.569797 | controller |       CINDER_KUTTL_NAMESPACE: cinder-kuttl-tests
2026-03-14 09:07:26.569803 | controller |       CINDER_REPO: https://github.com/openstack-k8s-operators/cinder-operator.git
2026-03-14 09:07:26.569809 | controller |       CLEANUP_DIR_CMD: rm -Rf
2026-03-14 09:07:26.569815 | controller |       CRC_BGP_NIC_1_MAC: '52:54:00:11:11:11'
2026-03-14 09:07:26.569821 | controller |       CRC_BGP_NIC_2_MAC: '52:54:00:11:11:12'
2026-03-14 09:07:26.569831 | controller |       CRC_HTTPS_PROXY: ''
2026-03-14 09:07:26.569837 | controller |       CRC_HTTP_PROXY: ''
2026-03-14 09:07:26.569844 | controller |       CRC_STORAGE_NAMESPACE: crc-storage
2026-03-14 09:07:26.569850 | controller |       CRC_STORAGE_RETRIES: '3'
2026-03-14 09:07:26.569856 | controller |       CRC_URL: '''https://developers.redhat.com/content-gateway/rest/mirror/pub/openshift-v4/clients/crc/latest/crc-linux-amd64.tar.xz'''
2026-03-14 09:07:26.569862 | controller |       CRC_VERSION: latest
2026-03-14 09:07:26.569868 | controller |       DATAPLANE_ANSIBLE_SECRET: dataplane-ansible-ssh-private-key-secret
2026-03-14 09:07:26.569874 | controller |       DATAPLANE_ANSIBLE_USER: ''
2026-03-14 09:07:26.569880 | controller |       DATAPLANE_COMPUTE_IP: 192.168.122.100
2026-03-14 09:07:26.569886 | controller |       DATAPLANE_CONTAINER_PREFIX: openstack
2026-03-14 09:07:26.569892 | controller |       DATAPLANE_CONTAINER_TAG: current-podified
2026-03-14 09:07:26.569898 | controller |       DATAPLANE_CUSTOM_SERVICE_RUNNER_IMG: quay.io/openstack-k8s-operators/openstack-ansibleee-runner:latest
2026-03-14 09:07:26.569904 | controller |       DATAPLANE_DEFAULT_GW: 192.168.122.1
2026-03-14 09:07:26.569910 | controller |       DATAPLANE_EXTRA_NOVA_CONFIG_FILE: /dev/null
2026-03-14 09:07:26.569916 | controller |       DATAPLANE_GROWVOLS_ARGS: /=8GB /tmp=1GB /home=1GB /var=100%
2026-03-14 09:07:26.569922 | controller |       DATAPLANE_KUSTOMIZE_SCENARIO: preprovisioned
2026-03-14 09:07:26.569928 | controller |       DATAPLANE_NETWORKER_IP: 192.168.122.200
2026-03-14 09:07:26.569934 | controller |       DATAPLANE_NETWORK_INTERFACE_NAME: eth0
2026-03-14 09:07:26.569940 | controller |       DATAPLANE_NOVA_NFS_PATH: ''
2026-03-14 09:07:26.569946 | controller |       DATAPLANE_NTP_SERVER: pool.ntp.org
2026-03-14 09:07:26.569951 | controller |       DATAPLANE_PLAYBOOK: osp.edpm.download_cache
2026-03-14 09:07:26.569957 | controller |       DATAPLANE_REGISTRY_URL: quay.io/podified-antelope-centos9
2026-03-14 09:07:26.569963 | controller |       DATAPLANE_RUNNER_IMG: ''
2026-03-14 09:07:26.569972 | controller |       DATAPLANE_SERVER_ROLE: compute
2026-03-14 09:07:26.569978 | controller |       DATAPLANE_SSHD_ALLOWED_RANGES: '[''192.168.122.0/24'']'
2026-03-14 09:07:26.569984 | controller |       DATAPLANE_TIMEOUT: 30m
2026-03-14 09:07:26.569990 | controller |       DATAPLANE_TLS_ENABLED: 'true'
2026-03-14 09:07:26.569996 | controller |       DATAPLANE_TOTAL_NETWORKER_NODES: '1'
2026-03-14 09:07:26.570002 | controller |       DATAPLANE_TOTAL_NODES: '1'
2026-03-14 09:07:26.570008 | controller |       DBSERVICE: galera
2026-03-14 09:07:26.570014 | controller |       DESIGNATE: config/samples/designate_v1beta1_designate.yaml
2026-03-14 09:07:26.570020 | controller |       DESIGNATE_BRANCH: 18.0-fr5
2026-03-14 09:07:26.570026 | controller |       DESIGNATE_COMMIT_HASH: ''
2026-03-14 09:07:26.570032 | controller |       DESIGNATE_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/designate-operator/config/samples/designate_v1beta1_designate.yaml
2026-03-14 09:07:26.570038 | controller |       DESIGNATE_IMG: quay.io/openstack-k8s-operators/designate-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.570044 | controller |       DESIGNATE_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/designate-operator/kuttl-test.yaml
2026-03-14 09:07:26.570050 | controller |       DESIGNATE_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/designate-operator/test/kuttl/tests
2026-03-14 09:07:26.570056 | controller |       DESIGNATE_KUTTL_NAMESPACE: designate-kuttl-tests
2026-03-14 09:07:26.570062 | controller |       DESIGNATE_REPO: https://github.com/openstack-k8s-operators/designate-operator.git
2026-03-14 09:07:26.570068 | controller |       DNSDATA: config/samples/network_v1beta1_dnsdata.yaml
2026-03-14 09:07:26.570077 | controller |       DNSDATA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/network_v1beta1_dnsdata.yaml
2026-03-14 09:07:26.570083 | controller |       DNSMASQ: config/samples/network_v1beta1_dnsmasq.yaml
2026-03-14 09:07:26.570089 | controller |       DNSMASQ_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/network_v1beta1_dnsmasq.yaml
2026-03-14 09:07:26.570095 | controller |       DNS_DEPL_IMG: unused
2026-03-14 09:07:26.570103 | controller |       DNS_DOMAIN: localdomain
2026-03-14 09:07:26.570110 | controller |       DOWNLOAD_TOOLS_SELECTION: all
2026-03-14 09:07:26.570116 | controller |       EDPM_ATTACH_EXTNET: 'true'
2026-03-14 09:07:26.570122 | controller |       EDPM_COMPUTE_ADDITIONAL_HOST_ROUTES: '''[]'''
2026-03-14 09:07:26.570130 | controller |       EDPM_COMPUTE_ADDITIONAL_NETWORKS: '''[]'''
2026-03-14 09:07:26.570136 | controller |       EDPM_COMPUTE_CELLS: '1'
2026-03-14 09:07:26.570143 | controller |       EDPM_COMPUTE_CEPH_ENABLED: 'true'
2026-03-14 09:07:26.570149 | controller |       EDPM_COMPUTE_CEPH_NOVA: 'true'
2026-03-14 09:07:26.570154 | controller |       EDPM_COMPUTE_DHCP_AGENT_ENABLED: 'true'
2026-03-14 09:07:26.570160 | controller |       EDPM_COMPUTE_SRIOV_ENABLED: 'true'
2026-03-14 09:07:26.570166 | controller |       EDPM_COMPUTE_SUFFIX: '0'
2026-03-14 09:07:26.570172 | controller |       EDPM_CONFIGURE_DEFAULT_ROUTE: 'true'
2026-03-14 09:07:26.570178 | controller |       EDPM_CONFIGURE_HUGEPAGES: 'false'
2026-03-14 09:07:26.570184 | controller |       EDPM_CONFIGURE_NETWORKING: 'true'
2026-03-14 09:07:26.570190 | controller |       EDPM_FIRSTBOOT_EXTRA: /tmp/edpm-firstboot-extra
2026-03-14 09:07:26.570196 | controller |       EDPM_NETWORKER_SUFFIX: '0'
2026-03-14 09:07:26.570202 | controller |       EDPM_TOTAL_NETWORKERS: '1'
2026-03-14 09:07:26.570208 | controller |       EDPM_TOTAL_NODES: '1'
2026-03-14 09:07:26.570214 | controller |       GALERA_REPLICAS: ''
2026-03-14 09:07:26.570220 | controller |       GENERATE_SSH_KEYS: 'true'
2026-03-14 09:07:26.570226 | controller |       GIT_CLONE_OPTS: ''
2026-03-14 09:07:26.570232 | controller |       GLANCE: config/samples/glance_v1beta1_glance.yaml
2026-03-14 09:07:26.570238 | controller |       GLANCEAPI_DEPL_IMG: unused
2026-03-14 09:07:26.570243 | controller |       GLANCE_BRANCH: 18.0-fr5
2026-03-14 09:07:26.570249 | controller |       GLANCE_COMMIT_HASH: ''
2026-03-14 09:07:26.570255 | controller |       GLANCE_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/glance-operator/config/samples/glance_v1beta1_glance.yaml
2026-03-14 09:07:26.570262 | controller |       GLANCE_IMG: quay.io/openstack-k8s-operators/glance-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.570268 | controller |       GLANCE_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/glance-operator/kuttl-test.yaml
2026-03-14 09:07:26.570277 | controller |       GLANCE_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/glance-operator/test/kuttl/tests
2026-03-14 09:07:26.570283 | controller |       GLANCE_KUTTL_NAMESPACE: glance-kuttl-tests
2026-03-14 09:07:26.570317 | controller |       GLANCE_REPO: https://github.com/openstack-k8s-operators/glance-operator.git
2026-03-14 09:07:26.570324 | controller |       HEAT: config/samples/heat_v1beta1_heat.yaml
2026-03-14 09:07:26.570330 | controller |       HEATAPI_DEPL_IMG: unused
2026-03-14 09:07:26.570336 | controller |       HEATCFNAPI_DEPL_IMG: unused
2026-03-14 09:07:26.570342 | controller |       HEATENGINE_DEPL_IMG: unused
2026-03-14 09:07:26.570348 | controller |       HEAT_AUTH_ENCRYPTION_KEY: 767c3ed056cbaa3b9dfedb8c6f825bf0
2026-03-14 09:07:26.570354 | controller |       HEAT_BRANCH: 18.0-fr5
2026-03-14 09:07:26.570360 | controller |       HEAT_COMMIT_HASH: ''
2026-03-14 09:07:26.570370 | controller |       HEAT_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/heat-operator/config/samples/heat_v1beta1_heat.yaml
2026-03-14 09:07:26.570376 | controller |       HEAT_IMG: quay.io/openstack-k8s-operators/heat-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.570382 | controller |       HEAT_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/heat-operator/kuttl-test.yaml
2026-03-14 09:07:26.570389 | controller |       HEAT_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/heat-operator/test/kuttl/tests
2026-03-14 09:07:26.570395 | controller |       HEAT_KUTTL_NAMESPACE: heat-kuttl-tests
2026-03-14 09:07:26.570401 | controller |       HEAT_REPO: https://github.com/openstack-k8s-operators/heat-operator.git
2026-03-14 09:07:26.570409 | controller |       HEAT_SERVICE_ENABLED: 'true'
2026-03-14 09:07:26.570415 | controller |       HORIZON: config/samples/horizon_v1beta1_horizon.yaml
2026-03-14 09:07:26.570420 | controller |       HORIZON_BRANCH: 18.0-fr5
2026-03-14 09:07:26.570426 | controller |       HORIZON_COMMIT_HASH: ''
2026-03-14 09:07:26.570432 | controller |       HORIZON_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/horizon-operator/config/samples/horizon_v1beta1_horizon.yaml
2026-03-14 09:07:26.570438 | controller |       HORIZON_DEPL_IMG: unused
2026-03-14 09:07:26.570444 | controller |       HORIZON_IMG: quay.io/openstack-k8s-operators/horizon-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.570450 | controller |       HORIZON_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/horizon-operator/kuttl-test.yaml
2026-03-14 09:07:26.570458 | controller |       HORIZON_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/horizon-operator/test/kuttl/tests
2026-03-14 09:07:26.570464 | controller |       HORIZON_KUTTL_NAMESPACE: horizon-kuttl-tests
2026-03-14 09:07:26.570470 | controller |       HORIZON_REPO: https://github.com/openstack-k8s-operators/horizon-operator.git
2026-03-14 09:07:26.570476 | controller |       INFRA_BRANCH: 18.0-fr5
2026-03-14 09:07:26.570482 | controller |       INFRA_COMMIT_HASH: ''
2026-03-14 09:07:26.570488 | controller |       INFRA_IMG: quay.io/openstack-k8s-operators/infra-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.570493 | controller |       INFRA_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/kuttl-test.yaml
2026-03-14 09:07:26.570500 | controller |       INFRA_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/test/kuttl/tests
2026-03-14 09:07:26.570506 | controller |       INFRA_KUTTL_NAMESPACE: infra-kuttl-tests
2026-03-14 09:07:26.570512 | controller |       INFRA_REPO: https://github.com/openstack-k8s-operators/infra-operator.git
2026-03-14 09:07:26.570518 | controller |       INSTALL_CERT_MANAGER: 'true'
2026-03-14 09:07:26.570523 | controller |       INSTALL_NMSTATE: true || false
2026-03-14 09:07:26.570529 | controller |       INSTALL_NNCP: true || false
2026-03-14 09:07:26.570535 | controller |       INTERNALAPI_HOST_ROUTES: ''
2026-03-14 09:07:26.570541 | controller |       IPV6_LAB_IPV4_NETWORK_IPADDRESS: 172.30.0.1/24
2026-03-14 09:07:26.570547 | controller |       IPV6_LAB_IPV6_NETWORK_IPADDRESS: fd00:abcd:abcd:fc00::1/64
2026-03-14 09:07:26.570553 | controller |       IPV6_LAB_LIBVIRT_STORAGE_POOL: default
2026-03-14 09:07:26.570559 | controller |       IPV6_LAB_MANAGE_FIREWALLD: 'true'
2026-03-14 09:07:26.570565 | controller |       IPV6_LAB_NAT64_HOST_IPV4: 172.30.0.2/24
2026-03-14 09:07:26.570571 | controller |       IPV6_LAB_NAT64_HOST_IPV6: fd00:abcd:abcd:fc00::2/64
2026-03-14 09:07:26.570583 | controller |       IPV6_LAB_NAT64_INSTANCE_NAME: nat64-router
2026-03-14 09:07:26.570593 | controller |       IPV6_LAB_NAT64_IPV6_NETWORK: fd00:abcd:abcd:fc00::/64
2026-03-14 09:07:26.570599 | controller |       IPV6_LAB_NAT64_TAYGA_DYNAMIC_POOL: 192.168.255.0/24
2026-03-14 09:07:26.570605 | controller |       IPV6_LAB_NAT64_TAYGA_IPV4: 192.168.255.1
2026-03-14 09:07:26.570613 | controller |       IPV6_LAB_NAT64_TAYGA_IPV6: fd00:abcd:abcd:fc00::3
2026-03-14 09:07:26.570619 | controller |       IPV6_LAB_NAT64_TAYGA_IPV6_PREFIX: fd00:abcd:abcd:fcff::/96
2026-03-14 09:07:26.570627 | controller |       IPV6_LAB_NAT64_UPDATE_PACKAGES: 'false'
2026-03-14 09:07:26.570633 | controller |       IPV6_LAB_NETWORK_NAME: nat64
2026-03-14 09:07:26.570640 | controller |       IPV6_LAB_SNO_CLUSTER_NETWORK: fd00:abcd:0::/48
2026-03-14 09:07:26.570648 | controller |       IPV6_LAB_SNO_HOST_IP: fd00:abcd:abcd:fc00::11
2026-03-14 09:07:26.570654 | controller |       IPV6_LAB_SNO_HOST_PREFIX: '64'
2026-03-14 09:07:26.570660 | controller |       IPV6_LAB_SNO_INSTANCE_NAME: sno
2026-03-14 09:07:26.570665 | controller |       IPV6_LAB_SNO_MACHINE_NETWORK: fd00:abcd:abcd:fc00::/64
2026-03-14 09:07:26.570671 | controller |       IPV6_LAB_SNO_OCP_MIRROR_URL: https://mirror.openshift.com/pub/openshift-v4/clients/ocp
2026-03-14 09:07:26.570677 | controller |       IPV6_LAB_SNO_OCP_VERSION: latest-4.14
2026-03-14 09:07:26.570683 | controller |       IPV6_LAB_SNO_SERVICE_NETWORK: fd00:abcd:abcd:fc03::/112
2026-03-14 09:07:26.570689 | controller |       IPV6_LAB_SSH_PUB_KEY: /home/zuul/.ssh/id_rsa.pub
2026-03-14 09:07:26.570695 | controller |       IPV6_LAB_WORK_DIR: /home/zuul/.ipv6lab
2026-03-14 09:07:26.570701 | controller |       IRONIC: config/samples/ironic_v1beta1_ironic.yaml
2026-03-14 09:07:26.570707 | controller |       IRONICAPI_DEPL_IMG: unused
2026-03-14 09:07:26.570713 | controller |       IRONICCON_DEPL_IMG: unused
2026-03-14 09:07:26.570719 | controller |       IRONICINS_DEPL_IMG: unused
2026-03-14 09:07:26.570728 | controller |       IRONICNAG_DEPL_IMG: unused
2026-03-14 09:07:26.570734 | controller |       IRONICPXE_DEPL_IMG: unused
2026-03-14 09:07:26.570740 | controller |       IRONIC_BRANCH: 18.0-fr5
2026-03-14 09:07:26.570746 | controller |       IRONIC_COMMIT_HASH: ''
2026-03-14 09:07:26.570752 | controller |       IRONIC_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ironic-operator/config/samples/ironic_v1beta1_ironic.yaml
2026-03-14 09:07:26.570758 | controller |       IRONIC_IMAGE: quay.io/metal3-io/ironic
2026-03-14 09:07:26.570764 | controller |       IRONIC_IMAGE_TAG: release-24.1
2026-03-14 09:07:26.570770 | controller |       IRONIC_IMG: quay.io/openstack-k8s-operators/ironic-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.570776 | controller |       IRONIC_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/ironic-operator/kuttl-test.yaml
2026-03-14 09:07:26.570782 | controller |       IRONIC_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ironic-operator/test/kuttl/tests
2026-03-14 09:07:26.570787 | controller |       IRONIC_KUTTL_NAMESPACE: ironic-kuttl-tests
2026-03-14 09:07:26.570793 | controller |       IRONIC_REPO: https://github.com/openstack-k8s-operators/ironic-operator.git
2026-03-14 09:07:26.570799 | controller |       KEYSTONEAPI: config/samples/keystone_v1beta1_keystoneapi.yaml
2026-03-14 09:07:26.570805 | controller |       KEYSTONEAPI_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/keystone-operator/config/samples/keystone_v1beta1_keystoneapi.yaml
2026-03-14 09:07:26.570813 | controller |       KEYSTONEAPI_DEPL_IMG: unused
2026-03-14 09:07:26.570819 | controller |       KEYSTONE_BRANCH: 18.0-fr5
2026-03-14 09:07:26.570825 | controller |       KEYSTONE_COMMIT_HASH: ''
2026-03-14 09:07:26.570831 | controller |       KEYSTONE_FEDERATION_CLIENT_SECRET: COX8bmlKAWn56XCGMrKQJj7dgHNAOl6f
2026-03-14 09:07:26.570837 | controller |       KEYSTONE_FEDERATION_CRYPTO_PASSPHRASE: openstack
2026-03-14 09:07:26.570843 | controller |       KEYSTONE_IMG: quay.io/openstack-k8s-operators/keystone-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.570849 | controller |       KEYSTONE_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/keystone-operator/kuttl-test.yaml
2026-03-14 09:07:26.570855 | controller |       KEYSTONE_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/keystone-operator/test/kuttl/tests
2026-03-14 09:07:26.570861 | controller |       KEYSTONE_KUTTL_NAMESPACE: keystone-kuttl-tests
2026-03-14 09:07:26.570867 | controller |       KEYSTONE_REPO: https://github.com/openstack-k8s-operators/keystone-operator.git
2026-03-14 09:07:26.570876 | controller |       KUBEADMIN_PWD: '12345678'
2026-03-14 09:07:26.570882 | controller |       LIBVIRT_SECRET: libvirt-secret
2026-03-14 09:07:26.570888 | controller |       LOKI_DEPLOY_MODE: openshift-network
2026-03-14 09:07:26.570894 | controller |       LOKI_DEPLOY_NAMESPACE: netobserv
2026-03-14 09:07:26.570899 | controller |       LOKI_DEPLOY_SIZE: 1x.demo
2026-03-14 09:07:26.570905 | controller |       LOKI_NAMESPACE: openshift-operators-redhat
2026-03-14 09:07:26.570911 | controller |       LOKI_OPERATOR_GROUP: openshift-operators-redhat-loki
2026-03-14 09:07:26.570917 | controller |       LOKI_SUBSCRIPTION: loki-operator
2026-03-14 09:07:26.570923 | controller |       LVMS_CR: '1'
2026-03-14 09:07:26.570929 | controller |       MANILA: config/samples/manila_v1beta1_manila.yaml
2026-03-14 09:07:26.570935 | controller |       MANILAAPI_DEPL_IMG: unused
2026-03-14 09:07:26.570941 | controller |       MANILASCH_DEPL_IMG: unused
2026-03-14 09:07:26.570947 | controller |       MANILASHARE_DEPL_IMG: unused
2026-03-14 09:07:26.570953 | controller |       MANILA_BRANCH: 18.0-fr5
2026-03-14 09:07:26.570959 | controller |       MANILA_COMMIT_HASH: ''
2026-03-14 09:07:26.570966 | controller |       MANILA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/manila-operator/config/samples/manila_v1beta1_manila.yaml
2026-03-14 09:07:26.570972 | controller |       MANILA_IMG: quay.io/openstack-k8s-operators/manila-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.570977 | controller |       MANILA_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/manila-operator/kuttl-test.yaml
2026-03-14 09:07:26.570983 | controller |       MANILA_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/manila-operator/test/kuttl/tests
2026-03-14 09:07:26.570989 | controller |       MANILA_KUTTL_NAMESPACE: manila-kuttl-tests
2026-03-14 09:07:26.570995 | controller |       MANILA_REPO: https://github.com/openstack-k8s-operators/manila-operator.git
2026-03-14 09:07:26.571002 | controller |       MANILA_SERVICE_ENABLED: 'true'
2026-03-14 09:07:26.571008 | controller |       MARIADB: config/samples/mariadb_v1beta1_galera.yaml
2026-03-14 09:07:26.571013 | controller |       MARIADB_BRANCH: 18.0-fr5
2026-03-14 09:07:26.571022 | controller |       MARIADB_CHAINSAW_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/test/chainsaw/config.yaml
2026-03-14 09:07:26.571028 | controller |       MARIADB_CHAINSAW_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/test/chainsaw/tests
2026-03-14 09:07:26.571035 | controller |       MARIADB_CHAINSAW_NAMESPACE: mariadb-chainsaw-tests
2026-03-14 09:07:26.571040 | controller |       MARIADB_COMMIT_HASH: ''
2026-03-14 09:07:26.571047 | controller |       MARIADB_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/config/samples/mariadb_v1beta1_galera.yaml
2026-03-14 09:07:26.571053 | controller |       MARIADB_DEPL_IMG: unused
2026-03-14 09:07:26.571059 | controller |       MARIADB_IMG: quay.io/openstack-k8s-operators/mariadb-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.571065 | controller |       MARIADB_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/kuttl-test.yaml
2026-03-14 09:07:26.571071 | controller |       MARIADB_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/test/kuttl/tests
2026-03-14 09:07:26.571077 | controller |       MARIADB_KUTTL_NAMESPACE: mariadb-kuttl-tests
2026-03-14 09:07:26.571083 | controller |       MARIADB_REPO: https://github.com/openstack-k8s-operators/mariadb-operator.git
2026-03-14 09:07:26.571088 | controller |       MEMCACHED: config/samples/memcached_v1beta1_memcached.yaml
2026-03-14 09:07:26.571095 | controller |       MEMCACHED_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/memcached_v1beta1_memcached.yaml
2026-03-14 09:07:26.571101 | controller |       MEMCACHED_DEPL_IMG: unused
2026-03-14 09:07:26.571109 | controller |       METADATA_SHARED_SECRET: '1234567842'
2026-03-14 09:07:26.571115 | controller |       METALLB_IPV6_POOL: fd00:aaaa::80-fd00:aaaa::90
2026-03-14 09:07:26.571121 | controller |       METALLB_POOL: 192.168.122.80-192.168.122.90
2026-03-14 09:07:26.571127 | controller |       MICROSHIFT: '0'
2026-03-14 09:07:26.571133 | controller |       MIRROR_INSECURE: 'true'
2026-03-14 09:07:26.571140 | controller |       MIRROR_NAMESPACE: openstack-mirror
2026-03-14 09:07:26.571146 | controller |       MIRROR_REGISTRY_HOST: default-route-openshift-image-registry.apps-crc.testing
2026-03-14 09:07:26.571154 | controller |       NAMESPACE: openstack
2026-03-14 09:07:26.571161 | controller |       NETCONFIG: config/samples/network_v1beta1_netconfig.yaml
2026-03-14 09:07:26.571167 | controller |       NETCONFIG_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/network_v1beta1_netconfig.yaml
2026-03-14 09:07:26.571173 | controller |       NETCONFIG_DEPL_IMG: unused
2026-03-14 09:07:26.571179 | controller |       NETOBSERV_DEPLOY_NAMESPACE: netobserv
2026-03-14 09:07:26.571185 | controller |       NETOBSERV_NAMESPACE: openshift-netobserv-operator
2026-03-14 09:07:26.571191 | controller |       NETOBSERV_OPERATOR_GROUP: openshift-netobserv-operator-net
2026-03-14 09:07:26.571197 | controller |       NETOBSERV_SUBSCRIPTION: netobserv-operator
2026-03-14 09:07:26.571203 | controller |       NETWORK_BGP: 'false'
2026-03-14 09:07:26.571209 | controller |       NETWORK_DESIGNATE_ADDRESS_PREFIX: 172.28.0
2026-03-14 09:07:26.571215 | controller |       NETWORK_DESIGNATE_EXT_ADDRESS_PREFIX: 172.50.0
2026-03-14 09:07:26.571221 | controller |       NETWORK_INTERNALAPI_ADDRESS_PREFIX: 172.17.0
2026-03-14 09:07:26.571228 | controller |       NETWORK_ISOLATION: 'true'
2026-03-14 09:07:26.571234 | controller |       NETWORK_ISOLATION_INSTANCE_NAME: crc
2026-03-14 09:07:26.571240 | controller |       NETWORK_ISOLATION_IPV4: 'true'
2026-03-14 09:07:26.571246 | controller |       NETWORK_ISOLATION_IPV4_ADDRESS: 172.16.1.1/24
2026-03-14 09:07:26.571252 | controller |       NETWORK_ISOLATION_IPV4_NAT: 'true'
2026-03-14 09:07:26.571258 | controller |       NETWORK_ISOLATION_IPV6: 'false'
2026-03-14 09:07:26.571264 | controller |       NETWORK_ISOLATION_IPV6_ADDRESS: fd00:aaaa::1/64
2026-03-14 09:07:26.571270 | controller |       NETWORK_ISOLATION_IP_ADDRESS: 192.168.122.10
2026-03-14 09:07:26.571276 | controller |       NETWORK_ISOLATION_MAC: '52:54:00:11:11:10'
2026-03-14 09:07:26.571282 | controller |       NETWORK_ISOLATION_NETWORK_NAME: net-iso
2026-03-14 09:07:26.571308 | controller |       NETWORK_ISOLATION_NET_NAME: default
2026-03-14 09:07:26.571315 | controller |       NETWORK_ISOLATION_USE_DEFAULT_NETWORK: 'true'
2026-03-14 09:07:26.571321 | controller |       NETWORK_MTU: '1500'
2026-03-14 09:07:26.571327 | controller |       NETWORK_STORAGEMGMT_ADDRESS_PREFIX: 172.20.0
2026-03-14 09:07:26.571333 | controller |       NETWORK_STORAGE_ADDRESS_PREFIX: 172.18.0
2026-03-14 09:07:26.571339 | controller |       NETWORK_STORAGE_MACVLAN: ''
2026-03-14 09:07:26.571346 | controller |       NETWORK_TENANT_ADDRESS_PREFIX: 172.19.0
2026-03-14 09:07:26.571352 | controller |       NETWORK_VLAN_START: '20'
2026-03-14 09:07:26.571358 | controller |       NETWORK_VLAN_STEP: '1'
2026-03-14 09:07:26.571364 | controller |       NEUTRONAPI: config/samples/neutron_v1beta1_neutronapi.yaml
2026-03-14 09:07:26.571370 | controller |       NEUTRONAPI_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/neutron-operator/config/samples/neutron_v1beta1_neutronapi.yaml
2026-03-14 09:07:26.571376 | controller |       NEUTRONAPI_DEPL_IMG: unused
2026-03-14 09:07:26.571383 | controller |       NEUTRON_BRANCH: 18.0-fr5
2026-03-14 09:07:26.571389 | controller |       NEUTRON_COMMIT_HASH: ''
2026-03-14 09:07:26.571395 | controller |       NEUTRON_IMG: quay.io/openstack-k8s-operators/neutron-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.571401 | controller |       NEUTRON_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/neutron-operator/kuttl-test.yaml
2026-03-14 09:07:26.571407 | controller |       NEUTRON_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/neutron-operator/test/kuttl/tests
2026-03-14 09:07:26.571413 | controller |       NEUTRON_KUTTL_NAMESPACE: neutron-kuttl-tests
2026-03-14 09:07:26.571419 | controller |       NEUTRON_REPO: https://github.com/openstack-k8s-operators/neutron-operator.git
2026-03-14 09:07:26.571427 | controller |       NFS_HOME: /home/nfs
2026-03-14 09:07:26.571433 | controller |       NMSTATE_NAMESPACE: openshift-nmstate
2026-03-14 09:07:26.571439 | controller |       NMSTATE_OPERATOR_GROUP: openshift-nmstate-tn6k8
2026-03-14 09:07:26.571445 | controller |       NMSTATE_SUBSCRIPTION: kubernetes-nmstate-operator
2026-03-14 09:07:26.571451 | controller |       NNCP_ADDITIONAL_HOST_ROUTES: ''
2026-03-14 09:07:26.571457 | controller |       NNCP_BGP_1_INTERFACE: enp7s0
2026-03-14 09:07:26.571464 | controller |       NNCP_BGP_1_IP_ADDRESS: 100.65.4.2
2026-03-14 09:07:26.571470 | controller |       NNCP_BGP_2_INTERFACE: enp8s0
2026-03-14 09:07:26.571476 | controller |       NNCP_BGP_2_IP_ADDRESS: 100.64.4.2
2026-03-14 09:07:26.571487 | controller |       NNCP_BRIDGE: ospbr
2026-03-14 09:07:26.571496 | controller |       NNCP_CLEANUP_TIMEOUT: 120s
2026-03-14 09:07:26.571502 | controller |       NNCP_CTLPLANE_IPV6_ADDRESS_PREFIX: 'fd00:aaaa::'
2026-03-14 09:07:26.571508 | controller |       NNCP_CTLPLANE_IPV6_ADDRESS_SUFFIX: '10'
2026-03-14 09:07:26.571515 | controller |       NNCP_CTLPLANE_IP_ADDRESS_PREFIX: 192.168.122
2026-03-14 09:07:26.571521 | controller |       NNCP_CTLPLANE_IP_ADDRESS_SUFFIX: '10'
2026-03-14 09:07:26.571527 | controller |       NNCP_DNS_SERVER: 192.168.122.1
2026-03-14 09:07:26.571533 | controller |       NNCP_DNS_SERVER_IPV6: fd00:aaaa::1
2026-03-14 09:07:26.571539 | controller |       NNCP_GATEWAY: 192.168.122.1
2026-03-14 09:07:26.571545 | controller |       NNCP_GATEWAY_IPV6: fd00:aaaa::1
2026-03-14 09:07:26.571551 | controller |       NNCP_INTERFACE: enp6s0
2026-03-14 09:07:26.571557 | controller |       NNCP_NODES: ''
2026-03-14 09:07:26.571564 | controller |       NNCP_TIMEOUT: 240s
2026-03-14 09:07:26.571570 | controller |       NOVA: config/samples/nova_v1beta1_nova_collapsed_cell.yaml
2026-03-14 09:07:26.571576 | controller |       NOVA_BRANCH: 18.0-fr5
2026-03-14 09:07:26.571582 | controller |       NOVA_COMMIT_HASH: ''
2026-03-14 09:07:26.571588 | controller |       NOVA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/nova-operator/config/samples/nova_v1beta1_nova_collapsed_cell.yaml
2026-03-14 09:07:26.571594 | controller |       NOVA_IMG: quay.io/openstack-k8s-operators/nova-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.571601 | controller |       NOVA_REPO: https://github.com/openstack-k8s-operators/nova-operator.git
2026-03-14 09:07:26.571607 | controller |       NUMBER_OF_INSTANCES: '1'
2026-03-14 09:07:26.571613 | controller |       OCP_NETWORK_NAME: crc
2026-03-14 09:07:26.571619 | controller |       OCTAVIA: config/samples/octavia_v1beta1_octavia.yaml
2026-03-14 09:07:26.571625 | controller |       OCTAVIA_BRANCH: 18.0-fr5
2026-03-14 09:07:26.571631 | controller |       OCTAVIA_COMMIT_HASH: ''
2026-03-14 09:07:26.571637 | controller |       OCTAVIA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/octavia-operator/config/samples/octavia_v1beta1_octavia.yaml
2026-03-14 09:07:26.571644 | controller |       OCTAVIA_IMG: quay.io/openstack-k8s-operators/octavia-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.571650 | controller |       OCTAVIA_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/octavia-operator/kuttl-test.yaml
2026-03-14 09:07:26.571656 | controller |       OCTAVIA_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/octavia-operator/test/kuttl/tests
2026-03-14 09:07:26.571662 | controller |       OCTAVIA_KUTTL_NAMESPACE: octavia-kuttl-tests
2026-03-14 09:07:26.571668 | controller |       OCTAVIA_REPO: https://github.com/openstack-k8s-operators/octavia-operator.git
2026-03-14 09:07:26.571674 | controller |       OKD: 'false'
2026-03-14 09:07:26.571680 | controller |       OPENSTACK_BRANCH: 18.0-fr5
2026-03-14 09:07:26.571686 | controller |       OPENSTACK_BUNDLE_IMG: quay.io/openstack-k8s-operators/openstack-operator-bundle:18.0-fr5-latest
2026-03-14 09:07:26.571693 | controller |       OPENSTACK_COMMIT_HASH: ''
2026-03-14 09:07:26.571699 | controller |       OPENSTACK_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/openstack-operator/config/samples/core_v1beta1_openstackcontrolplane_galera_network_isolation.yaml
2026-03-14 09:07:26.571705 | controller |       OPENSTACK_CRDS_DIR: openstack_crds
2026-03-14 09:07:26.571711 | controller |       OPENSTACK_CTLPLANE: config/samples/core_v1beta1_openstackcontrolplane_galera_network_isolation.yaml
2026-03-14 09:07:26.571717 | controller |       OPENSTACK_IMG: quay.io/openstack-k8s-operators/openstack-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.571724 | controller |       OPENSTACK_K8S_BRANCH: 18.0-fr5
2026-03-14 09:07:26.571730 | controller |       OPENSTACK_K8S_TAG: 18.0-fr5-latest
2026-03-14 09:07:26.571739 | controller |       OPENSTACK_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/openstack-operator/kuttl-test.yaml
2026-03-14 09:07:26.571745 | controller |       OPENSTACK_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/openstack-operator/test/kuttl/tests
2026-03-14 09:07:26.571751 | controller |       OPENSTACK_KUTTL_NAMESPACE: openstack-kuttl-tests
2026-03-14 09:07:26.571757 | controller |       OPENSTACK_NEUTRON_CUSTOM_CONF: ''
2026-03-14 09:07:26.571764 | controller |       OPENSTACK_REPO: https://github.com/openstack-k8s-operators/openstack-operator.git
2026-03-14 09:07:26.571772 | controller |       OPENSTACK_STORAGE_BUNDLE_IMG: quay.io/openstack-k8s-operators/openstack-operator-storage-bundle:18.0-fr5-latest
2026-03-14 09:07:26.571779 | controller |       OPERATOR_BASE_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator
2026-03-14 09:07:26.571785 | controller |       OPERATOR_CHANNEL: ''
2026-03-14 09:07:26.571791 | controller |       OPERATOR_NAMESPACE: openstack-operators
2026-03-14 09:07:26.571797 | controller |       OPERATOR_SOURCE: ''
2026-03-14 09:07:26.571803 | controller |       OPERATOR_SOURCE_NAMESPACE: ''
2026-03-14 09:07:26.571809 | controller |       OUT: /home/zuul/ci-framework-data/artifacts/manifests
2026-03-14 09:07:26.571816 | controller |       OUTPUT_DIR: /home/zuul/ci-framework-data/artifacts/edpm
2026-03-14 09:07:26.571822 | controller |       OVNCONTROLLER: config/samples/ovn_v1beta1_ovncontroller.yaml
2026-03-14 09:07:26.571830 | controller |       OVNCONTROLLER_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/config/samples/ovn_v1beta1_ovncontroller.yaml
2026-03-14 09:07:26.571836 | controller |       OVNCONTROLLER_NMAP: 'true'
2026-03-14 09:07:26.571842 | controller |       OVNDBS: config/samples/ovn_v1beta1_ovndbcluster.yaml
2026-03-14 09:07:26.571849 | controller |       OVNDBS_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/config/samples/ovn_v1beta1_ovndbcluster.yaml
2026-03-14 09:07:26.571907 | controller |       OVNNORTHD: config/samples/ovn_v1beta1_ovnnorthd.yaml
2026-03-14 09:07:26.571921 | controller |       OVNNORTHD_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/config/samples/ovn_v1beta1_ovnnorthd.yaml
2026-03-14 09:07:26.571927 | controller |       OVN_BRANCH: 18.0-fr5
2026-03-14 09:07:26.571933 | controller |       OVN_COMMIT_HASH: ''
2026-03-14 09:07:26.571940 | controller |       OVN_IMG: quay.io/openstack-k8s-operators/ovn-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.571949 | controller |       OVN_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/kuttl-test.yaml
2026-03-14 09:07:26.572012 | controller |       OVN_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/test/kuttl/tests
2026-03-14 09:07:26.572023 | controller |       OVN_KUTTL_NAMESPACE: ovn-kuttl-tests
2026-03-14 09:07:26.572032 | controller |       OVN_REPO: https://github.com/openstack-k8s-operators/ovn-operator.git
2026-03-14 09:07:26.572038 | controller |       PASSWORD: '12345678'
2026-03-14 09:07:26.572048 | controller |       PLACEMENTAPI: config/samples/placement_v1beta1_placementapi.yaml
2026-03-14 09:07:26.572054 | controller |       PLACEMENTAPI_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/placement-operator/config/samples/placement_v1beta1_placementapi.yaml
2026-03-14 09:07:26.572061 | controller |       PLACEMENTAPI_DEPL_IMG: unused
2026-03-14 09:07:26.572067 | controller |       PLACEMENT_BRANCH: 18.0-fr5
2026-03-14 09:07:26.572073 | controller |       PLACEMENT_COMMIT_HASH: ''
2026-03-14 09:07:26.572079 | controller |       PLACEMENT_IMG: quay.io/openstack-k8s-operators/placement-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.572085 | controller |       PLACEMENT_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/placement-operator/kuttl-test.yaml
2026-03-14 09:07:26.572091 | controller |       PLACEMENT_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/placement-operator/test/kuttl/tests
2026-03-14 09:07:26.572098 | controller |       PLACEMENT_KUTTL_NAMESPACE: placement-kuttl-tests
2026-03-14 09:07:26.572104 | controller |       PLACEMENT_REPO: https://github.com/openstack-k8s-operators/placement-operator.git
2026-03-14 09:07:26.572110 | controller |       PULL_SECRET: /home/zuul/pull-secret.txt
2026-03-14 09:07:26.572116 | controller |       RABBITMQ: docs/examples/default-security-context/rabbitmq.yaml
2026-03-14 09:07:26.572122 | controller |       RABBITMQ_BRANCH: patches
2026-03-14 09:07:26.572128 | controller |       RABBITMQ_COMMIT_HASH: ''
2026-03-14 09:07:26.572134 | controller |       RABBITMQ_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/rabbitmq-operator/docs/examples/default-security-context/rabbitmq.yaml
2026-03-14 09:07:26.572141 | controller |       RABBITMQ_DEPL_IMG: unused
2026-03-14 09:07:26.572147 | controller |       RABBITMQ_IMG: quay.io/openstack-k8s-operators/rabbitmq-cluster-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.572153 | controller |       RABBITMQ_REPO: https://github.com/openstack-k8s-operators/rabbitmq-cluster-operator.git
2026-03-14 09:07:26.572162 | controller |       REDHAT_OPERATORS: 'false'
2026-03-14 09:07:26.572168 | controller |       REDIS: config/samples/redis_v1beta1_redis.yaml
2026-03-14 09:07:26.572174 | controller |       REDIS_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator-redis/config/samples/redis_v1beta1_redis.yaml
2026-03-14 09:07:26.572180 | controller |       REDIS_DEPL_IMG: unused
2026-03-14 09:07:26.572187 | controller |       RH_REGISTRY_PWD: ''
2026-03-14 09:07:26.572193 | controller |       RH_REGISTRY_USER: ''
2026-03-14 09:07:26.572199 | controller |       SECRET: osp-secret
2026-03-14 09:07:26.572249 | controller |       SG_CORE_DEPL_IMG: unused
2026-03-14 09:07:26.572264 | controller |       STANDALONE_COMPUTE_DRIVER: libvirt
2026-03-14 09:07:26.572271 | controller |       STANDALONE_EXTERNAL_NET_PREFFIX: 172.21.0
2026-03-14 09:07:26.572278 | controller |       STANDALONE_INTERNALAPI_NET_PREFIX: 172.17.0
2026-03-14 09:07:26.572351 | controller |       STANDALONE_STORAGEMGMT_NET_PREFIX: 172.20.0
2026-03-14 09:07:26.572404 | controller |       STANDALONE_STORAGE_NET_PREFIX: 172.18.0
2026-03-14 09:07:26.572413 | controller |       STANDALONE_TENANT_NET_PREFIX: 172.19.0
2026-03-14 09:07:26.572419 | controller |       STORAGEMGMT_HOST_ROUTES: ''
2026-03-14 09:07:26.572425 | controller |       STORAGE_CLASS: local-storage
2026-03-14 09:07:26.572431 | controller |       STORAGE_HOST_ROUTES: ''
2026-03-14 09:07:26.572460 | controller |       SWIFT: config/samples/swift_v1beta1_swift.yaml
2026-03-14 09:07:26.572466 | controller |       SWIFT_BRANCH: 18.0-fr5
2026-03-14 09:07:26.572472 | controller |       SWIFT_COMMIT_HASH: ''
2026-03-14 09:07:26.572478 | controller |       SWIFT_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/swift-operator/config/samples/swift_v1beta1_swift.yaml
2026-03-14 09:07:26.572484 | controller |       SWIFT_IMG: quay.io/openstack-k8s-operators/swift-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.572490 | controller |       SWIFT_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/swift-operator/kuttl-test.yaml
2026-03-14 09:07:26.572496 | controller |       SWIFT_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/swift-operator/test/kuttl/tests
2026-03-14 09:07:26.572502 | controller |       SWIFT_KUTTL_NAMESPACE: swift-kuttl-tests
2026-03-14 09:07:26.572508 | controller |       SWIFT_REPO: https://github.com/openstack-k8s-operators/swift-operator.git
2026-03-14 09:07:26.572514 | controller |       TELEMETRY: config/samples/telemetry_v1beta1_telemetry.yaml
2026-03-14 09:07:26.572524 | controller |       TELEMETRY_BRANCH: 18.0-fr5
2026-03-14 09:07:26.572530 | controller |       TELEMETRY_COMMIT_HASH: ''
2026-03-14 09:07:26.572536 | controller |       TELEMETRY_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator/config/samples/telemetry_v1beta1_telemetry.yaml
2026-03-14 09:07:26.572542 | controller |       TELEMETRY_IMG: quay.io/openstack-k8s-operators/telemetry-operator-index:18.0-fr5-latest
2026-03-14 09:07:26.572548 | controller |       TELEMETRY_KUTTL_BASEDIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator
2026-03-14 09:07:26.572554 | controller |       TELEMETRY_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator/kuttl-test.yaml
2026-03-14 09:07:26.572560 | controller |       TELEMETRY_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator/test/kuttl/suites
2026-03-14 09:07:26.572566 | controller |       TELEMETRY_KUTTL_NAMESPACE: telemetry-kuttl-tests
2026-03-14 09:07:26.572572 | controller |       TELEMETRY_KUTTL_RELPATH: test/kuttl/suites
2026-03-14 09:07:26.572578 | controller |       TELEMETRY_REPO: https://github.com/openstack-k8s-operators/telemetry-operator.git
2026-03-14 09:07:26.572584 | controller |       TENANT_HOST_ROUTES: ''
2026-03-14 09:07:26.572590 | controller |       TIMEOUT: 300s
2026-03-14 09:07:26.572597 | controller |       TLS_ENABLED: 'false'
2026-03-14 09:07:26.572605 | controller |       WATCHER_BRANCH: ''
2026-03-14 09:07:26.572612 | controller |       WATCHER_REPO: /home/zuul/src/github.com/openstack-k8s-operators/watcher-operator
2026-03-14 09:07:26.572622 | controller |       tripleo_deploy: 'export REGISTRY_USER:'
2026-03-14 09:07:26.579084 | controller |
2026-03-14 09:07:26.929352 | controller | TASK [install_yamls : Generate make targets install_yamls_path={{ cifmw_install_yamls_repo }}, output_directory={{ cifmw_install_yamls_tasks_out }}] ***
2026-03-14 09:07:26.929386 | controller | Saturday 14 March 2026  09:07:26 +0000 (0:00:00.037)       0:01:29.327 ********
2026-03-14 09:07:26.929397 | controller | Saturday 14 March 2026  09:07:26 +0000 (0:00:00.037)       0:01:29.325 ********
2026-03-14 09:07:26.929406 | controller | ok: [controller]
2026-03-14 09:07:26.946719 | controller |
2026-03-14 09:07:26.946776 | controller | TASK [install_yamls : Debug generate_make module var=cifmw_generate_makes] *****
2026-03-14 09:07:26.946783 | controller | Saturday 14 March 2026  09:07:26 +0000 (0:00:00.350)       0:01:29.677 ********
2026-03-14 09:07:26.946790 | controller | Saturday 14 March 2026  09:07:26 +0000 (0:00:00.350)       0:01:29.676 ********
2026-03-14 09:07:26.946804 | controller | ok: [controller] =>
2026-03-14 09:07:26.946812 | controller |     cifmw_generate_makes:
2026-03-14 09:07:26.946817 | controller |       changed: false
2026-03-14 09:07:26.946824 | controller |       debug:
2026-03-14 09:07:26.946830 | controller |         /home/zuul/src/github.com/openstack-k8s-operators/install_yamls/Makefile:
2026-03-14 09:07:26.946835 | controller |         - all
2026-03-14 09:07:26.946841 | controller |         - help
2026-03-14 09:07:26.946847 | controller |         - cleanup
2026-03-14 09:07:26.946853 | controller |         - deploy_cleanup
2026-03-14 09:07:26.946859 | controller |         - wait
2026-03-14 09:07:26.946865 | controller |         - crc_storage
2026-03-14 09:07:26.946871 | controller |         - crc_storage_cleanup
2026-03-14 09:07:26.946877 | controller |         - crc_storage_release
2026-03-14 09:07:26.946883 | controller |         - crc_storage_with_retries
2026-03-14 09:07:26.946890 | controller |         - crc_storage_cleanup_with_retries
2026-03-14 09:07:26.946896 | controller |         - operator_namespace
2026-03-14 09:07:26.946902 | controller |         - namespace
2026-03-14 09:07:26.946908 | controller |         - namespace_cleanup
2026-03-14 09:07:26.946914 | controller |         - input
2026-03-14 09:07:26.946920 | controller |         - input_cleanup
2026-03-14 09:07:26.946926 | controller |         - crc_bmo_setup
2026-03-14 09:07:26.946931 | controller |         - crc_bmo_cleanup
2026-03-14 09:07:26.946938 | controller |         - mirror_registry
2026-03-14 09:07:26.946944 | controller |         - mirror_registry_secure
2026-03-14 09:07:26.946950 | controller |         - mirror_registry_ca
2026-03-14 09:07:26.946956 | controller |         - mirror_registry_cleanup
2026-03-14 09:07:26.946961 | controller |         - openstack_prep
2026-03-14 09:07:26.946968 | controller |         - openstack
2026-03-14 09:07:26.946974 | controller |         - openstack_wait
2026-03-14 09:07:26.946980 | controller |         - openstack_init
2026-03-14 09:07:26.946986 | controller |         - openstack_cleanup
2026-03-14 09:07:26.946991 | controller |         - openstack_repo
2026-03-14 09:07:26.946997 | controller |         - openstack_deploy_prep
2026-03-14 09:07:26.947003 | controller |         - openstack_deploy
2026-03-14 09:07:26.947009 | controller |         - openstack_wait_deploy
2026-03-14 09:07:26.947015 | controller |         - openstack_deploy_cleanup
2026-03-14 09:07:26.947021 | controller |         - openstack_update_run
2026-03-14 09:07:26.947027 | controller |         - update_services
2026-03-14 09:07:26.947034 | controller |         - update_system
2026-03-14 09:07:26.947040 | controller |         - openstack_patch_version
2026-03-14 09:07:26.947046 | controller |         - edpm_deploy_generate_keys
2026-03-14 09:07:26.947052 | controller |         - edpm_patch_ansible_runner_image
2026-03-14 09:07:26.947058 | controller |         - edpm_deploy_prep
2026-03-14 09:07:26.947064 | controller |         - edpm_deploy_cleanup
2026-03-14 09:07:26.947070 | controller |         - edpm_deploy
2026-03-14 09:07:26.947076 | controller |         - edpm_deploy_baremetal_prep
2026-03-14 09:07:26.947082 | controller |         - edpm_deploy_baremetal
2026-03-14 09:07:26.947088 | controller |         - edpm_wait_deploy_baremetal
2026-03-14 09:07:26.947094 | controller |         - edpm_wait_deploy
2026-03-14 09:07:26.947101 | controller |         - edpm_register_dns
2026-03-14 09:07:26.947107 | controller |         - edpm_nova_discover_hosts
2026-03-14 09:07:26.947113 | controller |         - openstack_crds
2026-03-14 09:07:26.947119 | controller |         - openstack_crds_cleanup
2026-03-14 09:07:26.947126 | controller |         - edpm_deploy_networker_prep
2026-03-14 09:07:26.947132 | controller |         - edpm_deploy_networker_cleanup
2026-03-14 09:07:26.947138 | controller |         - edpm_deploy_networker
2026-03-14 09:07:26.947144 | controller |         - infra_prep
2026-03-14 09:07:26.947151 | controller |         - infra
2026-03-14 09:07:26.947157 | controller |         - infra_cleanup
2026-03-14 09:07:26.947163 | controller |         - dns_deploy_prep
2026-03-14 09:07:26.947169 | controller |         - dns_deploy
2026-03-14 09:07:26.947175 | controller |         - dns_deploy_cleanup
2026-03-14 09:07:26.947187 | controller |         - netconfig_deploy_prep
2026-03-14 09:07:26.947194 | controller |         - netconfig_deploy
2026-03-14 09:07:26.947201 | controller |         - netconfig_deploy_cleanup
2026-03-14 09:07:26.947207 | controller |         - memcached_deploy_prep
2026-03-14 09:07:26.947213 | controller |         - memcached_deploy
2026-03-14 09:07:26.947219 | controller |         - memcached_deploy_cleanup
2026-03-14 09:07:26.947226 | controller |         - keystone_prep
2026-03-14 09:07:26.947232 | controller |         - keystone
2026-03-14 09:07:26.947238 | controller |         - keystone_cleanup
2026-03-14 09:07:26.947244 | controller |         - keystone_deploy_prep
2026-03-14 09:07:26.947251 | controller |         - keystone_deploy
2026-03-14 09:07:26.947257 | controller |         - keystone_deploy_cleanup
2026-03-14 09:07:26.947263 | controller |         - barbican_prep
2026-03-14 09:07:26.947269 | controller |         - barbican
2026-03-14 09:07:26.947276 | controller |         - barbican_cleanup
2026-03-14 09:07:26.947282 | controller |         - barbican_deploy_prep
2026-03-14 09:07:26.947317 | controller |         - barbican_deploy
2026-03-14 09:07:26.947324 | controller |         - barbican_deploy_validate
2026-03-14 09:07:26.947331 | controller |         - barbican_deploy_cleanup
2026-03-14 09:07:26.947337 | controller |         - mariadb
2026-03-14 09:07:26.947343 | controller |         - mariadb_cleanup
2026-03-14 09:07:26.947350 | controller |         - mariadb_deploy_prep
2026-03-14 09:07:26.947356 | controller |         - mariadb_deploy
2026-03-14 09:07:26.947362 | controller |         - mariadb_deploy_cleanup
2026-03-14 09:07:26.947368 | controller |         - placement_prep
2026-03-14 09:07:26.947374 | controller |         - placement
2026-03-14 09:07:26.947381 | controller |         - placement_cleanup
2026-03-14 09:07:26.947387 | controller |         - placement_deploy_prep
2026-03-14 09:07:26.947393 | controller |         - placement_deploy
2026-03-14 09:07:26.947399 | controller |         - placement_deploy_cleanup
2026-03-14 09:07:26.947406 | controller |         - glance_prep
2026-03-14 09:07:26.947419 | controller |         - glance
2026-03-14 09:07:26.947425 | controller |         - glance_cleanup
2026-03-14 09:07:26.947431 | controller |         - glance_deploy_prep
2026-03-14 09:07:26.947438 | controller |         - glance_deploy
2026-03-14 09:07:26.947444 | controller |         - glance_deploy_cleanup
2026-03-14 09:07:26.947450 | controller |         - ovn_prep
2026-03-14 09:07:26.947456 | controller |         - ovn
2026-03-14 09:07:26.947463 | controller |         - ovn_cleanup
2026-03-14 09:07:26.947469 | controller |         - ovn_deploy_prep
2026-03-14 09:07:26.947475 | controller |         - ovn_deploy
2026-03-14 09:07:26.947481 | controller |         - ovn_deploy_cleanup
2026-03-14 09:07:26.947488 | controller |         - neutron_prep
2026-03-14 09:07:26.947494 | controller |         - neutron
2026-03-14 09:07:26.947500 | controller |         - neutron_cleanup
2026-03-14 09:07:26.947506 | controller |         - neutron_deploy_prep
2026-03-14 09:07:26.947513 | controller |         - neutron_deploy
2026-03-14 09:07:26.947519 | controller |         - neutron_deploy_cleanup
2026-03-14 09:07:26.947529 | controller |         - cinder_prep
2026-03-14 09:07:26.947536 | controller |         - cinder
2026-03-14 09:07:26.947543 | controller |         - cinder_cleanup
2026-03-14 09:07:26.947549 | controller |         - cinder_deploy_prep
2026-03-14 09:07:26.947555 | controller |         - cinder_deploy
2026-03-14 09:07:26.947562 | controller |         - cinder_deploy_cleanup
2026-03-14 09:07:26.947568 | controller |         - rabbitmq_prep
2026-03-14 09:07:26.947575 | controller |         - rabbitmq
2026-03-14 09:07:26.947581 | controller |         - rabbitmq_cleanup
2026-03-14 09:07:26.947587 | controller |         - rabbitmq_deploy_prep
2026-03-14 09:07:26.947593 | controller |         - rabbitmq_deploy
2026-03-14 09:07:26.947599 | controller |         - rabbitmq_deploy_cleanup
2026-03-14 09:07:26.947606 | controller |         - ironic_prep
2026-03-14 09:07:26.947612 | controller |         - ironic
2026-03-14 09:07:26.947617 | controller |         - ironic_cleanup
2026-03-14 09:07:26.947623 | controller |         - ironic_deploy_prep
2026-03-14 09:07:26.947629 | controller |         - ironic_deploy
2026-03-14 09:07:26.947635 | controller |         - ironic_deploy_cleanup
2026-03-14 09:07:26.947641 | controller |         - octavia_prep
2026-03-14 09:07:26.947647 | controller |         - octavia
2026-03-14 09:07:26.947653 | controller |         - octavia_cleanup
2026-03-14 09:07:26.947659 | controller |         - octavia_deploy_prep
2026-03-14 09:07:26.947665 | controller |         - octavia_deploy
2026-03-14 09:07:26.947671 | controller |         - octavia_deploy_cleanup
2026-03-14 09:07:26.947677 | controller |         - designate_prep
2026-03-14 09:07:26.947683 | controller |         - designate
2026-03-14 09:07:26.947693 | controller |         - designate_cleanup
2026-03-14 09:07:26.947699 | controller |         - designate_deploy_prep
2026-03-14 09:07:26.947707 | controller |         - designate_deploy
2026-03-14 09:07:26.947713 | controller |         - designate_deploy_cleanup
2026-03-14 09:07:26.947721 | controller |         - nova_prep
2026-03-14 09:07:26.947728 | controller |         - nova
2026-03-14 09:07:26.947734 | controller |         - nova_cleanup
2026-03-14 09:07:26.947740 | controller |         - nova_deploy_prep
2026-03-14 09:07:26.947746 | controller |         - nova_deploy
2026-03-14 09:07:26.947752 | controller |         - nova_deploy_cleanup
2026-03-14 09:07:26.947758 | controller |         - mariadb_kuttl_run
2026-03-14 09:07:26.947764 | controller |         - mariadb_kuttl
2026-03-14 09:07:26.947770 | controller |         - kuttl_db_prep
2026-03-14 09:07:26.947776 | controller |         - kuttl_db_cleanup
2026-03-14 09:07:26.947782 | controller |         - kuttl_common_prep
2026-03-14 09:07:26.947794 | controller |         - kuttl_common_cleanup
2026-03-14 09:07:26.947801 | controller |         - keystone_kuttl_run
2026-03-14 09:07:26.947807 | controller |         - keystone_kuttl
2026-03-14 09:07:26.947813 | controller |         - barbican_kuttl_run
2026-03-14 09:07:26.947819 | controller |         - barbican_kuttl
2026-03-14 09:07:26.947825 | controller |         - placement_kuttl_run
2026-03-14 09:07:26.947831 | controller |         - placement_kuttl
2026-03-14 09:07:26.947837 | controller |         - cinder_kuttl_run
2026-03-14 09:07:26.947843 | controller |         - cinder_kuttl
2026-03-14 09:07:26.947849 | controller |         - neutron_kuttl_run
2026-03-14 09:07:26.947854 | controller |         - neutron_kuttl
2026-03-14 09:07:26.947861 | controller |         - octavia_kuttl_run
2026-03-14 09:07:26.947867 | controller |         - octavia_kuttl
2026-03-14 09:07:26.947873 | controller |         - designate_kuttl
2026-03-14 09:07:26.947879 | controller |         - designate_kuttl_run
2026-03-14 09:07:26.947887 | controller |         - ovn_kuttl_run
2026-03-14 09:07:26.947893 | controller |         - ovn_kuttl
2026-03-14 09:07:26.947899 | controller |         - infra_kuttl_run
2026-03-14 09:07:26.947905 | controller |         - infra_kuttl
2026-03-14 09:07:26.947911 | controller |         - ironic_kuttl_run
2026-03-14 09:07:26.947916 | controller |         - ironic_kuttl
2026-03-14 09:07:26.947922 | controller |         - ironic_kuttl_crc
2026-03-14 09:07:26.947928 | controller |         - heat_kuttl_run
2026-03-14 09:07:26.947934 | controller |         - heat_kuttl
2026-03-14 09:07:26.947940 | controller |         - heat_kuttl_crc
2026-03-14 09:07:26.947946 | controller |         - glance_kuttl_run
2026-03-14 09:07:26.947952 | controller |         - glance_kuttl
2026-03-14 09:07:26.947958 | controller |         - manila_kuttl_run
2026-03-14 09:07:26.947964 | controller |         - manila_kuttl
2026-03-14 09:07:26.947970 | controller |         - swift_kuttl_run
2026-03-14 09:07:26.947976 | controller |         - swift_kuttl
2026-03-14 09:07:26.947982 | controller |         - horizon_kuttl_run
2026-03-14 09:07:26.947988 | controller |         - horizon_kuttl
2026-03-14 09:07:26.947994 | controller |         - openstack_kuttl_prep
2026-03-14 09:07:26.948000 | controller |         - openstack_kuttl_run
2026-03-14 09:07:26.948006 | controller |         - openstack_kuttl_cleanup
2026-03-14 09:07:26.948012 | controller |         - openstack_kuttl
2026-03-14 09:07:26.948018 | controller |         - mariadb_chainsaw_run
2026-03-14 09:07:26.948024 | controller |         - mariadb_chainsaw
2026-03-14 09:07:26.948030 | controller |         - horizon_prep
2026-03-14 09:07:26.948036 | controller |         - horizon
2026-03-14 09:07:26.948042 | controller |         - horizon_cleanup
2026-03-14 09:07:26.948048 | controller |         - horizon_deploy_prep
2026-03-14 09:07:26.948054 | controller |         - horizon_deploy
2026-03-14 09:07:26.948059 | controller |         - horizon_deploy_cleanup
2026-03-14 09:07:26.948065 | controller |         - heat_prep
2026-03-14 09:07:26.948071 | controller |         - heat
2026-03-14 09:07:26.948077 | controller |         - heat_cleanup
2026-03-14 09:07:26.948082 | controller |         - heat_deploy_prep
2026-03-14 09:07:26.948088 | controller |         - heat_deploy
2026-03-14 09:07:26.948094 | controller |         - heat_deploy_cleanup
2026-03-14 09:07:26.948100 | controller |         - baremetal_prep
2026-03-14 09:07:26.948105 | controller |         - baremetal
2026-03-14 09:07:26.948111 | controller |         - baremetal_cleanup
2026-03-14 09:07:26.948117 | controller |         - ceph_help
2026-03-14 09:07:26.948123 | controller |         - ceph
2026-03-14 09:07:26.948128 | controller |         - ceph_cleanup
2026-03-14 09:07:26.948134 | controller |         - rook_prep
2026-03-14 09:07:26.948140 | controller |         - rook
2026-03-14 09:07:26.948146 | controller |         - rook_deploy_prep
2026-03-14 09:07:26.948151 | controller |         - rook_deploy
2026-03-14 09:07:26.948160 | controller |         - rook_crc_disk
2026-03-14 09:07:26.948166 | controller |         - rook_cleanup
2026-03-14 09:07:26.948171 | controller |         - lvms
2026-03-14 09:07:26.948323 | controller |         - nmstate
2026-03-14 09:07:26.948339 | controller |         - nncp
2026-03-14 09:07:26.948345 | controller |         - nncp_cleanup
2026-03-14 09:07:26.948349 | controller |         - netattach
2026-03-14 09:07:26.948353 | controller |         - netattach_cleanup
2026-03-14 09:07:26.948358 | controller |         - metallb
2026-03-14 09:07:26.948362 | controller |         - metallb_config
2026-03-14 09:07:26.948366 | controller |         - metallb_config_cleanup
2026-03-14 09:07:26.948370 | controller |         - metallb_cleanup
2026-03-14 09:07:26.948374 | controller |         - loki
2026-03-14 09:07:26.948379 | controller |         - loki_cleanup
2026-03-14 09:07:26.948383 | controller |         - loki_deploy
2026-03-14 09:07:26.948387 | controller |         - loki_deploy_cleanup
2026-03-14 09:07:26.948391 | controller |         - netobserv
2026-03-14 09:07:26.948395 | controller |         - netobserv_cleanup
2026-03-14 09:07:26.948400 | controller |         - netobserv_deploy
2026-03-14 09:07:26.948404 | controller |         - netobserv_deploy_cleanup
2026-03-14 09:07:26.948408 | controller |         - manila_prep
2026-03-14 09:07:26.948413 | controller |         - manila
2026-03-14 09:07:26.948417 | controller |         - manila_cleanup
2026-03-14 09:07:26.948421 | controller |         - manila_deploy_prep
2026-03-14 09:07:26.948426 | controller |         - manila_deploy
2026-03-14 09:07:26.948430 | controller |         - manila_deploy_cleanup
2026-03-14 09:07:26.948434 | controller |         - telemetry_prep
2026-03-14 09:07:26.948439 | controller |         - telemetry
2026-03-14 09:07:26.948445 | controller |         - telemetry_cleanup
2026-03-14 09:07:26.948449 | controller |         - telemetry_deploy_prep
2026-03-14 09:07:26.948454 | controller |         - telemetry_deploy
2026-03-14 09:07:26.948458 | controller |         - telemetry_deploy_cleanup
2026-03-14 09:07:26.948462 | controller |         - telemetry_kuttl_run
2026-03-14 09:07:26.948467 | controller |         - telemetry_kuttl
2026-03-14 09:07:26.948471 | controller |         - swift_prep
2026-03-14 09:07:26.948475 | controller |         - swift
2026-03-14 09:07:26.948480 | controller |         - swift_cleanup
2026-03-14 09:07:26.948484 | controller |         - swift_deploy_prep
2026-03-14 09:07:26.948488 | controller |         - swift_deploy
2026-03-14 09:07:26.948493 | controller |         - swift_deploy_cleanup
2026-03-14 09:07:26.948497 | controller |         - certmanager
2026-03-14 09:07:26.948501 | controller |         - certmanager_cleanup
2026-03-14 09:07:26.948506 | controller |         - validate_marketplace
2026-03-14 09:07:26.948510 | controller |         - redis_deploy_prep
2026-03-14 09:07:26.948514 | controller |         - redis_deploy
2026-03-14 09:07:26.948518 | controller |         - redis_deploy_cleanup
2026-03-14 09:07:26.948523 | controller |         - set_slower_etcd_profile
2026-03-14 09:07:26.948532 | controller |         /home/zuul/src/github.com/openstack-k8s-operators/install_yamls/devsetup/Makefile:
2026-03-14 09:07:26.948537 | controller |         - help
2026-03-14 09:07:26.948542 | controller |         - download_tools
2026-03-14 09:07:26.948547 | controller |         - nfs
2026-03-14 09:07:26.948552 | controller |         - nfs_cleanup
2026-03-14 09:07:26.948557 | controller |         - crc
2026-03-14 09:07:26.948562 | controller |         - crc_cleanup
2026-03-14 09:07:26.948567 | controller |         - crc_scrub
2026-03-14 09:07:26.948572 | controller |         - crc_attach_default_interface
2026-03-14 09:07:26.948577 | controller |         - crc_attach_default_interface_cleanup
2026-03-14 09:07:26.948582 | controller |         - ipv6_lab_network
2026-03-14 09:07:26.948587 | controller |         - ipv6_lab_network_cleanup
2026-03-14 09:07:26.948592 | controller |         - ipv6_lab_nat64_router
2026-03-14 09:07:26.948597 | controller |         - ipv6_lab_nat64_router_cleanup
2026-03-14 09:07:26.948602 | controller |         - ipv6_lab_sno
2026-03-14 09:07:26.948606 | controller |         - ipv6_lab_sno_cleanup
2026-03-14 09:07:26.948610 | controller |         - ipv6_lab
2026-03-14 09:07:26.948615 | controller |         - ipv6_lab_cleanup
2026-03-14 09:07:26.948619 | controller |         - attach_default_interface
2026-03-14 09:07:26.948623 | controller |         - attach_default_interface_cleanup
2026-03-14 09:07:26.948628 | controller |         - network_isolation_bridge
2026-03-14 09:07:26.948632 | controller |         - network_isolation_bridge_cleanup
2026-03-14 09:07:26.948636 | controller |         - edpm_baremetal_compute
2026-03-14 09:07:26.948641 | controller |         - edpm_compute
2026-03-14 09:07:26.948645 | controller |         - edpm_compute_bootc
2026-03-14 09:07:26.948652 | controller |         - edpm_ansible_runner
2026-03-14 09:07:26.948656 | controller |         - edpm_computes_bgp
2026-03-14 09:07:26.948660 | controller |         - edpm_compute_repos
2026-03-14 09:07:26.948665 | controller |         - edpm_compute_cleanup
2026-03-14 09:07:26.948669 | controller |         - edpm_networker
2026-03-14 09:07:26.948673 | controller |         - edpm_networker_cleanup
2026-03-14 09:07:26.948678 | controller |         - edpm_deploy_instance
2026-03-14 09:07:26.948683 | controller |         - tripleo_deploy
2026-03-14 09:07:26.948687 | controller |         - standalone_deploy
2026-03-14 09:07:26.948692 | controller |         - standalone_sync
2026-03-14 09:07:26.948697 | controller |         - standalone
2026-03-14 09:07:26.948701 | controller |         - standalone_cleanup
2026-03-14 09:07:26.948706 | controller |         - standalone_snapshot
2026-03-14 09:07:26.948710 | controller |         - standalone_revert
2026-03-14 09:07:26.948715 | controller |         - cifmw_prepare
2026-03-14 09:07:26.948719 | controller |         - cifmw_cleanup
2026-03-14 09:07:26.948724 | controller |         - bmaas_network
2026-03-14 09:07:26.948729 | controller |         - bmaas_network_cleanup
2026-03-14 09:07:26.948733 | controller |         - bmaas_route_crc_and_crc_bmaas_networks
2026-03-14 09:07:26.948738 | controller |         - bmaas_route_crc_and_crc_bmaas_networks_cleanup
2026-03-14 09:07:26.948747 | controller |         - bmaas_crc_attach_network
2026-03-14 09:07:26.956098 | controller |         - bmaas_crc_attach_network_cleanup
2026-03-14 09:07:26.956116 | controller |         - bmaas_crc_baremetal_bridge
2026-03-14 09:07:26.956122 | controller |         - bmaas_crc_baremetal_bridge_cleanup
2026-03-14 09:07:26.956155 | controller |         - bmaas_baremetal_net_nad
2026-03-14 09:07:26.956163 | controller |         - bmaas_baremetal_net_nad_cleanup
2026-03-14 09:07:26.956167 | controller |         - bmaas_metallb
2026-03-14 09:07:26.956171 | controller |         - bmaas_metallb_cleanup
2026-03-14 09:07:26.956175 | controller |         - bmaas_virtual_bms
2026-03-14 09:07:26.956180 | controller |         - bmaas_virtual_bms_cleanup
2026-03-14 09:07:26.956184 | controller |         - bmaas_sushy_emulator
2026-03-14 09:07:26.956188 | controller |         - bmaas_sushy_emulator_cleanup
2026-03-14 09:07:26.956218 | controller |         - bmaas_sushy_emulator_wait
2026-03-14 09:07:26.956225 | controller |         - bmaas_generate_nodes_yaml
2026-03-14 09:07:26.956229 | controller |         - bmaas
2026-03-14 09:07:26.956233 | controller |         - bmaas_cleanup
2026-03-14 09:07:26.956238 | controller |       failed: false
2026-03-14 09:07:26.956242 | controller |       success: true
2026-03-14 09:07:26.956250 | controller |
2026-03-14 09:07:28.174567 | controller | TASK [install_yamls : Create the install_yamls parameters file dest={{ cifmw_basedir }}/artifacts/parameters/install-yamls-params.yml, content={{
2026-03-14 09:07:28.174598 | controller |   {
2026-03-14 09:07:28.174604 | controller |   'cifmw_install_yamls_environment': cifmw_install_yamls_environment,
2026-03-14 09:07:28.174609 | controller |   'cifmw_install_yamls_defaults': cifmw_install_yamls_defaults
2026-03-14 09:07:28.174614 | controller |   } | to_nice_yaml
2026-03-14 09:07:28.174619 | controller | }}, mode=0644] ***
2026-03-14 09:07:28.174623 | controller | Saturday 14 March 2026  09:07:26 +0000 (0:00:00.026)       0:01:29.704 ********
2026-03-14 09:07:28.174628 | controller | Saturday 14 March 2026  09:07:26 +0000 (0:00:00.026)       0:01:29.702 ********
2026-03-14 09:07:28.174637 | controller | changed: [controller]
2026-03-14 09:07:28.202871 | controller |
2026-03-14 09:07:28.202904 | controller | TASK [install_yamls : Create empty cifmw_install_yamls_environment if needed cifmw_install_yamls_environment={}] ***
2026-03-14 09:07:28.202911 | controller | Saturday 14 March 2026  09:07:28 +0000 (0:00:01.218)       0:01:30.922 ********
2026-03-14 09:07:28.202916 | controller | Saturday 14 March 2026  09:07:28 +0000 (0:00:01.218)       0:01:30.921 ********
2026-03-14 09:07:28.202925 | controller | skipping: [controller]
2026-03-14 09:07:28.222116 | controller |
2026-03-14 09:07:28.222147 | controller | TASK [discover_latest_image : Set the discovered image vars to empty/null cifmw_discovered_images_dict={}, cifmw_discovered_image_name=None, cifmw_discovered_image_url=None, cifmw_discovered_hash=None, cifmw_discovered_hash_algorithm=None] ***
2026-03-14 09:07:28.222154 | controller | Saturday 14 March 2026  09:07:28 +0000 (0:00:00.028)       0:01:30.951 ********
2026-03-14 09:07:28.222159 | controller | Saturday 14 March 2026  09:07:28 +0000 (0:00:00.028)       0:01:30.949 ********
2026-03-14 09:07:28.222168 | controller | ok: [controller]
2026-03-14 09:07:28.252923 | controller |
2026-03-14 09:07:28.252982 | controller | TASK [discover_latest_image : Discover images _raw_params=per-image.yml] *******
2026-03-14 09:07:28.252989 | controller | Saturday 14 March 2026  09:07:28 +0000 (0:00:00.019)       0:01:30.970 ********
2026-03-14 09:07:28.252994 | controller | Saturday 14 March 2026  09:07:28 +0000 (0:00:00.019)       0:01:30.968 ********
2026-03-14 09:07:28.253007 | controller | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/discover_latest_image/tasks/per-image.yml for controller => (item={'key': 'default', 'value': {'base_url': 'https://cloud.centos.org/centos/9-stream/x86_64/images/', 'qcow_prefix': 'CentOS-Stream-GenericCloud-', 'images_file': 'CHECKSUM'}})
2026-03-14 09:07:28.287588 | controller |
2026-03-14 09:07:28.287664 | controller | TASK [discover_latest_image : Ensure the current image dict entry has the expected data that=['_expected_keys | difference(_given_keys) | length == 0'], fail_msg=The image discovery request dict's {{ _current_image.key }} entry does not match the expected schema. Requests must have the following keys: {{ _expected_keys }}, this entry had: {{ _given_keys }} See the discover_latest_image role's README for more information.] ***
2026-03-14 09:07:28.287677 | controller | Saturday 14 March 2026  09:07:28 +0000 (0:00:00.030)       0:01:31.001 ********
2026-03-14 09:07:28.287683 | controller | Saturday 14 March 2026  09:07:28 +0000 (0:00:00.030)       0:01:30.999 ********
2026-03-14 09:07:28.287694 | controller | ok: [controller] =>
2026-03-14 09:07:28.993417 | controller |     changed: false
2026-03-14 09:07:28.993469 | controller |     msg: All assertions passed
2026-03-14 09:07:28.993482 | controller |
2026-03-14 09:07:28.993493 | controller | TASK [discover_latest_image : Get latest image for the current entry url={{ _current_image.value.base_url }}, image_prefix={{ _current_image.value.qcow_prefix }}, images_file={{ _current_image.value.images_file }}] ***
2026-03-14 09:07:28.993505 | controller | Saturday 14 March 2026  09:07:28 +0000 (0:00:00.034)       0:01:31.035 ********
2026-03-14 09:07:28.993515 | controller | Saturday 14 March 2026  09:07:28 +0000 (0:00:00.034)       0:01:31.034 ********
2026-03-14 09:07:28.993533 | controller | changed: [controller]
2026-03-14 09:07:29.025559 | controller |
2026-03-14 09:07:29.025625 | controller | TASK [discover_latest_image : Append the current return value to the discovered images dict cifmw_discovered_images_dict={{ cifmw_discovered_images_dict | combine({_current_image.key: discovered_image.data}) }}, cacheable=True] ***
2026-03-14 09:07:29.025641 | controller | Saturday 14 March 2026  09:07:28 +0000 (0:00:00.705)       0:01:31.740 ********
2026-03-14 09:07:29.025652 | controller | Saturday 14 March 2026  09:07:28 +0000 (0:00:00.705)       0:01:31.739 ********
2026-03-14 09:07:29.025670 | controller | ok: [controller]
2026-03-14 09:07:29.082423 | controller |
2026-03-14 09:07:29.082512 | controller | TASK [discover_latest_image : Export single-image facts if an image with name 'default' was discovered cifmw_discovered_image_name={{ cifmw_discovered_images_dict.default.image_name }}, cifmw_discovered_image_url={{ cifmw_discovered_images_dict.default.image_url }}, cifmw_discovered_hash={{ cifmw_discovered_images_dict.default.hash }}, cifmw_discovered_hash_algorithm={{ cifmw_discovered_images_dict.default.hash_algorithm }}, cacheable=True] ***
2026-03-14 09:07:29.082534 | controller | Saturday 14 March 2026  09:07:29 +0000 (0:00:00.032)       0:01:31.773 ********
2026-03-14 09:07:29.082552 | controller | Saturday 14 March 2026  09:07:29 +0000 (0:00:00.032)       0:01:31.772 ********
2026-03-14 09:07:29.082581 | controller | ok: [controller]
2026-03-14 09:07:29.797711 | controller |
2026-03-14 09:07:29.797768 | controller | TASK [cifmw_setup : Create artifacts with custom params mode=0644, dest={{ cifmw_basedir }}/artifacts/parameters/custom-params.yml, content={{ ci_framework_params | to_nice_yaml }}] ***
2026-03-14 09:07:29.797775 | controller | Saturday 14 March 2026  09:07:29 +0000 (0:00:00.056)       0:01:31.830 ********
2026-03-14 09:07:29.797781 | controller | Saturday 14 March 2026  09:07:29 +0000 (0:00:00.056)       0:01:31.829 ********
2026-03-14 09:07:29.797793 | controller | changed: [controller]
2026-03-14 09:07:29.822031 | controller |
2026-03-14 09:07:29.822109 | controller | PLAY [Install dev tools] *******************************************************
2026-03-14 09:07:29.822130 | controller |
2026-03-14 09:07:29.822141 | controller | TASK [Assert that operator_name is set that=['operator_name is defined']] ******
2026-03-14 09:07:29.822151 | controller | Saturday 14 March 2026  09:07:29 +0000 (0:00:00.715)       0:01:32.545 ********
2026-03-14 09:07:29.822161 | controller | Saturday 14 March 2026  09:07:29 +0000 (0:00:00.715)       0:01:32.544 ********
2026-03-14 09:07:29.822180 | controller | ok: [controller] =>
2026-03-14 09:07:29.851856 | controller |     changed: false
2026-03-14 09:07:29.851909 | controller |     msg: All assertions passed
2026-03-14 09:07:29.851916 | controller |
2026-03-14 09:07:29.851921 | controller | TASK [Download install_yamls deps name=install_yamls_makes, tasks_from=make_download_tools] ***
2026-03-14 09:07:29.851926 | controller | Saturday 14 March 2026  09:07:29 +0000 (0:00:00.024)       0:01:32.570 ********
2026-03-14 09:07:29.851931 | controller | Saturday 14 March 2026  09:07:29 +0000 (0:00:00.024)       0:01:32.568 ********
2026-03-14 09:07:29.851942 | controller |
2026-03-14 09:07:29.871756 | controller | TASK [install_yamls_makes : Debug make_download_tools_env var=make_download_tools_env] ***
2026-03-14 09:07:29.871806 | controller | Saturday 14 March 2026  09:07:29 +0000 (0:00:00.029)       0:01:32.599 ********
2026-03-14 09:07:29.871812 | controller | Saturday 14 March 2026  09:07:29 +0000 (0:00:00.029)       0:01:32.598 ********
2026-03-14 09:07:29.871822 | controller | skipping: [controller]
2026-03-14 09:07:29.893216 | controller |
2026-03-14 09:07:29.893260 | controller | TASK [install_yamls_makes : Debug make_download_tools_params var=make_download_tools_params] ***
2026-03-14 09:07:29.893267 | controller | Saturday 14 March 2026  09:07:29 +0000 (0:00:00.019)       0:01:32.619 ********
2026-03-14 09:07:29.893272 | controller | Saturday 14 March 2026  09:07:29 +0000 (0:00:00.019)       0:01:32.618 ********
2026-03-14 09:07:29.893282 | controller | skipping: [controller]
2026-03-14 09:08:09.163673 | controller |
2026-03-14 09:08:09.163746 | controller | TASK [install_yamls_makes : Run download_tools output_dir={{ cifmw_basedir }}/artifacts, chdir=/home/zuul/src/github.com/openstack-k8s-operators/install_yamls/devsetup, script=make download_tools, dry_run={{ make_download_tools_dryrun|default(false)|bool }}, extra_args={{ dict((make_download_tools_env|default({})), **(make_download_tools_params|default({}))) }}] ***
2026-03-14 09:08:09.163762 | controller | Saturday 14 March 2026  09:07:29 +0000 (0:00:00.021)       0:01:32.641 ********
2026-03-14 09:08:09.163774 | controller | Saturday 14 March 2026  09:07:29 +0000 (0:00:00.021)       0:01:32.640 ********
2026-03-14 09:08:09.163793 | controller | [WARNING]: conditional statements should not include jinja2 templating
2026-03-14 09:08:09.180670 | controller | delimiters such as {{ }} or {% %}. Found: {{ make_download_tools_until |
2026-03-14 09:08:09.180749 | controller | default(true) }}
2026-03-14 09:08:09.180780 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_000_run_download.log
2026-03-14 09:08:09.271443 | controller | changed: [controller]
2026-03-14 09:08:09.271498 | controller |
2026-03-14 09:08:09.271504 | controller | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] ***
2026-03-14 09:08:09.271510 | controller | Saturday 14 March 2026  09:08:09 +0000 (0:00:39.287)       0:02:11.928 ********
2026-03-14 09:08:09.271515 | controller | Saturday 14 March 2026  09:08:09 +0000 (0:00:39.287)       0:02:11.927 ********
2026-03-14 09:08:09.271526 | controller | ok: [controller]
2026-03-14 09:08:09.374445 | controller |
2026-03-14 09:08:09.374485 | controller | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] ***
2026-03-14 09:08:09.374494 | controller | Saturday 14 March 2026  09:08:09 +0000 (0:00:00.090)       0:02:12.019 ********
2026-03-14 09:08:09.374501 | controller | Saturday 14 March 2026  09:08:09 +0000 (0:00:00.090)       0:02:12.018 ********
2026-03-14 09:08:09.374514 | controller | ok: [controller]
2026-03-14 09:08:09.507529 | controller |
2026-03-14 09:08:09.507571 | controller | TASK [run_hook : Loop on hooks for pre_infra _raw_params={{ hook.type }}.yml] ***
2026-03-14 09:08:09.507580 | controller | Saturday 14 March 2026  09:08:09 +0000 (0:00:00.102)       0:02:12.122 ********
2026-03-14 09:08:09.507593 | controller | Saturday 14 March 2026  09:08:09 +0000 (0:00:00.102)       0:02:12.121 ********
2026-03-14 09:08:09.507607 | controller | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/run_hook/tasks/playbook.yml for controller => (item={'name': 'Download needed tools', 'inventory': 'localhost,', 'connection': 'local', 'type': 'playbook', 'source': '/home/zuul/src/github.com/openstack-k8s-operators/install_yamls/devsetup/download_tools.yaml'})
2026-03-14 09:08:09.560463 | controller |
2026-03-14 09:08:09.560496 | controller | TASK [run_hook : Set playbook path for Download needed tools cifmw_basedir={{ _bdir }}, hook_name={{ _hook_name }}, playbook_path={{ _play | realpath }}, log_path={{ _bdir }}/logs/{{ step }}_{{ _hook_name }}.log, extra_vars=-e namespace={{ cifmw_openstack_namespace }} {%- if hook.extra_vars is defined and hook.extra_vars|length > 0 -%} {% for key,value in hook.extra_vars.items() -%} {%- if key == 'file' %} -e "@{{ value }}" {%- else %} -e "{{ key }}={{ value }}" {%- endif %} {%- endfor %} {%- endif %}] ***
2026-03-14 09:08:09.560504 | controller | Saturday 14 March 2026  09:08:09 +0000 (0:00:00.132)       0:02:12.255 ********
2026-03-14 09:08:09.560509 | controller | Saturday 14 March 2026  09:08:09 +0000 (0:00:00.133)       0:02:12.254 ********
2026-03-14 09:08:09.560519 | controller | ok: [controller]
2026-03-14 09:08:09.781103 | controller |
2026-03-14 09:08:09.781138 | controller | TASK [run_hook : Get file stat path={{ playbook_path }}] ***********************
2026-03-14 09:08:09.781144 | controller | Saturday 14 March 2026  09:08:09 +0000 (0:00:00.052)       0:02:12.308 ********
2026-03-14 09:08:09.781149 | controller | Saturday 14 March 2026  09:08:09 +0000 (0:00:00.052)       0:02:12.307 ********
2026-03-14 09:08:09.781159 | controller | ok: [controller]
2026-03-14 09:08:09.800188 | controller |
2026-03-14 09:08:09.800238 | controller | TASK [run_hook : Fail if playbook doesn't exist msg=Playbook {{ playbook_path }} doesn't seem to exist.] ***
2026-03-14 09:08:09.800245 | controller | Saturday 14 March 2026  09:08:09 +0000 (0:00:00.220)       0:02:12.528 ********
2026-03-14 09:08:09.800250 | controller | Saturday 14 March 2026  09:08:09 +0000 (0:00:00.220)       0:02:12.527 ********
2026-03-14 09:08:09.800262 | controller | skipping: [controller]
2026-03-14 09:08:09.986435 | controller |
2026-03-14 09:08:09.986468 | controller | TASK [run_hook : Get parameters files paths={{
2026-03-14 09:08:09.986474 | controller |   (cifmw_basedir, 'artifacts/parameters') | path_join
2026-03-14 09:08:09.986480 | controller | }}, file_type=file, patterns=*.yml] ***
2026-03-14 09:08:09.986484 | controller | Saturday 14 March 2026  09:08:09 +0000 (0:00:00.019)       0:02:12.548 ********
2026-03-14 09:08:09.986489 | controller | Saturday 14 March 2026  09:08:09 +0000 (0:00:00.019)       0:02:12.546 ********
2026-03-14 09:08:09.986498 | controller | ok: [controller]
2026-03-14 09:08:10.009503 | controller |
2026-03-14 09:08:10.009535 | controller | TASK [run_hook : Add parameters artifacts as extra variables extra_vars={{ extra_vars }} {% for file in cifmw_run_hook_parameters_files.files %} -e "@{{ file.path }}" {%- endfor %}] ***
2026-03-14 09:08:10.009541 | controller | Saturday 14 March 2026  09:08:09 +0000 (0:00:00.186)       0:02:12.734 ********
2026-03-14 09:08:10.009546 | controller | Saturday 14 March 2026  09:08:09 +0000 (0:00:00.186)       0:02:12.733 ********
2026-03-14 09:08:10.009556 | controller | ok: [controller]
2026-03-14 09:08:10.212391 | controller |
2026-03-14 09:08:10.212424 | controller | TASK [run_hook : Ensure log directory exists path={{ log_path | dirname }}, state=directory, mode=0755] ***
2026-03-14 09:08:10.212430 | controller | Saturday 14 March 2026  09:08:10 +0000 (0:00:00.023)       0:02:12.757 ********
2026-03-14 09:08:10.212439 | controller | Saturday 14 March 2026  09:08:10 +0000 (0:00:00.023)       0:02:12.756 ********
2026-03-14 09:08:10.212448 | controller | ok: [controller]
2026-03-14 09:08:10.454676 | controller |
2026-03-14 09:08:10.454742 | controller | TASK [run_hook : Ensure artifacts directory exists path={{ cifmw_basedir }}/artifacts, state=directory, mode=0755] ***
2026-03-14 09:08:10.454760 | controller | Saturday 14 March 2026  09:08:10 +0000 (0:00:00.202)       0:02:12.960 ********
2026-03-14 09:08:10.454770 | controller | Saturday 14 March 2026  09:08:10 +0000 (0:00:00.202)       0:02:12.959 ********
2026-03-14 09:08:10.454794 | controller | ok: [controller]
2026-03-14 09:08:44.262263 | controller |
2026-03-14 09:08:44.262323 | controller | TASK [run_hook : Run hook without retry - Download needed tools] ***************
2026-03-14 09:08:44.262332 | controller | Saturday 14 March 2026  09:08:10 +0000 (0:00:00.241)       0:02:13.201 ********
2026-03-14 09:08:44.262336 | controller | Saturday 14 March 2026  09:08:10 +0000 (0:00:00.241)       0:02:13.200 ********
2026-03-14 09:08:44.262349 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_001_run_hook_without_retry.log
2026-03-14 09:08:44.283198 | controller | changed: [controller]
2026-03-14 09:08:44.283246 | controller |
2026-03-14 09:08:44.283252 | controller | TASK [run_hook : Run hook with retry - Download needed tools] ******************
2026-03-14 09:08:44.283257 | controller | Saturday 14 March 2026  09:08:44 +0000 (0:00:33.808)       0:02:47.010 ********
2026-03-14 09:08:44.283261 | controller | Saturday 14 March 2026  09:08:44 +0000 (0:00:33.808)       0:02:47.009 ********
2026-03-14 09:08:44.283274 | controller | skipping: [controller]
2026-03-14 09:08:44.490232 | controller |
2026-03-14 09:08:44.490281 | controller | TASK [run_hook : Check if we have a file path={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] ***
2026-03-14 09:08:44.490301 | controller | Saturday 14 March 2026  09:08:44 +0000 (0:00:00.020)       0:02:47.031 ********
2026-03-14 09:08:44.490308 | controller | Saturday 14 March 2026  09:08:44 +0000 (0:00:00.020)       0:02:47.030 ********
2026-03-14 09:08:44.490319 | controller | ok: [controller]
2026-03-14 09:08:44.577160 | controller |
2026-03-14 09:08:44.577216 | controller | TASK [run_hook : Load generated content in main playbook file={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] ***
2026-03-14 09:08:44.577223 | controller | Saturday 14 March 2026  09:08:44 +0000 (0:00:00.206)       0:02:47.237 ********
2026-03-14 09:08:44.577228 | controller | Saturday 14 March 2026  09:08:44 +0000 (0:00:00.206)       0:02:47.236 ********
2026-03-14 09:08:44.577239 | controller | skipping: [controller]
2026-03-14 09:08:44.659917 | controller |
2026-03-14 09:08:44.659974 | controller | PLAY [Prepare host virtualization] *********************************************
2026-03-14 09:08:44.659981 | controller |
2026-03-14 09:08:44.659986 | controller | TASK [cifmw_setup : Load parameters files dir={{ cifmw_basedir }}/artifacts/parameters] ***
2026-03-14 09:08:44.659991 | controller | Saturday 14 March 2026  09:08:44 +0000 (0:00:00.087)       0:02:47.325 ********
2026-03-14 09:08:44.659995 | controller | Saturday 14 March 2026  09:08:44 +0000 (0:00:00.087)       0:02:47.324 ********
2026-03-14 09:08:44.660007 | controller | ok: [controller]
2026-03-14 09:08:44.686624 | controller |
2026-03-14 09:08:44.686673 | controller | TASK [Ensure libvirt is present/configured name=libvirt_manager] ***************
2026-03-14 09:08:44.686680 | controller | Saturday 14 March 2026  09:08:44 +0000 (0:00:00.082)       0:02:47.407 ********
2026-03-14 09:08:44.686685 | controller | Saturday 14 March 2026  09:08:44 +0000 (0:00:00.082)       0:02:47.406 ********
2026-03-14 09:08:44.686695 | controller | skipping: [controller]
2026-03-14 09:08:44.768475 | controller |
2026-03-14 09:08:44.768545 | controller | TASK [Perpare OpenShift provisioner node name=openshift_provisioner_node] ******
2026-03-14 09:08:44.768559 | controller | Saturday 14 March 2026  09:08:44 +0000 (0:00:00.026)       0:02:47.434 ********
2026-03-14 09:08:44.768569 | controller | Saturday 14 March 2026  09:08:44 +0000 (0:00:00.026)       0:02:47.433 ********
2026-03-14 09:08:44.768588 | controller | skipping: [controller]
2026-03-14 09:08:44.890169 | controller |
2026-03-14 09:08:44.890229 | controller | PLAY [Build dataset hook] ******************************************************
2026-03-14 09:08:44.890236 | controller |
2026-03-14 09:08:44.890241 | controller | TASK [cifmw_setup : Load parameters files dir={{ cifmw_basedir }}/artifacts/parameters] ***
2026-03-14 09:08:44.890246 | controller | Saturday 14 March 2026  09:08:44 +0000 (0:00:00.081)       0:02:47.516 ********
2026-03-14 09:08:44.890250 | controller | Saturday 14 March 2026  09:08:44 +0000 (0:00:00.081)       0:02:47.515 ********
2026-03-14 09:08:44.890262 | controller | ok: [controller]
2026-03-14 09:08:45.100047 | controller |
2026-03-14 09:08:45.100114 | controller | TASK [networking_mapper : Check for Networking Environment Definition file existence path={{ cifmw_networking_mapper_networking_env_def_path }}] ***
2026-03-14 09:08:45.100125 | controller | Saturday 14 March 2026  09:08:44 +0000 (0:00:00.121)       0:02:47.638 ********
2026-03-14 09:08:45.100141 | controller | Saturday 14 March 2026  09:08:44 +0000 (0:00:00.121)       0:02:47.636 ********
2026-03-14 09:08:45.100160 | controller | ok: [controller]
2026-03-14 09:08:45.128188 | controller |
2026-03-14 09:08:45.128251 | controller | TASK [networking_mapper : Check for Networking Definition file existance that=['_net_env_def_stat.stat.exists'], msg=Ensure that the Networking Environment Definition file exists in {{ cifmw_networking_mapper_networking_env_def_path }}, quiet=True] ***
2026-03-14 09:08:45.128261 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.209)       0:02:47.848 ********
2026-03-14 09:08:45.128269 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.209)       0:02:47.846 ********
2026-03-14 09:08:45.128330 | controller | skipping: [controller]
2026-03-14 09:08:45.157750 | controller |
2026-03-14 09:08:45.157803 | controller | TASK [networking_mapper : Load the Networking Definition from file path={{ cifmw_networking_mapper_networking_env_def_path }}] ***
2026-03-14 09:08:45.157810 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.028)       0:02:47.876 ********
2026-03-14 09:08:45.157815 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.028)       0:02:47.875 ********
2026-03-14 09:08:45.157828 | controller | skipping: [controller]
2026-03-14 09:08:45.191448 | controller |
2026-03-14 09:08:45.191509 | controller | TASK [networking_mapper : Set cifmw_networking_env_definition is present cifmw_networking_env_definition={{
2026-03-14 09:08:45.191519 | controller |   _net_env_def_slurp['content'] |
2026-03-14 09:08:45.191526 | controller |   b64decode |
2026-03-14 09:08:45.191533 | controller |   from_yaml
2026-03-14 09:08:45.191540 | controller | }}, cacheable=True] ***
2026-03-14 09:08:45.191546 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.029)       0:02:47.905 ********
2026-03-14 09:08:45.191553 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.029)       0:02:47.904 ********
2026-03-14 09:08:45.191569 | controller | skipping: [controller]
2026-03-14 09:08:45.220373 | controller |
2026-03-14 09:08:45.220411 | controller | TASK [Deploy OCP using Hive name=hive] *****************************************
2026-03-14 09:08:45.220420 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.033)       0:02:47.939 ********
2026-03-14 09:08:45.220427 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.033)       0:02:47.938 ********
2026-03-14 09:08:45.220439 | controller | skipping: [controller]
2026-03-14 09:08:45.248270 | controller |
2026-03-14 09:08:45.248340 | controller | TASK [Prepare CRC name=rhol_crc] ***********************************************
2026-03-14 09:08:45.248351 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.028)       0:02:47.968 ********
2026-03-14 09:08:45.248358 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.028)       0:02:47.967 ********
2026-03-14 09:08:45.248371 | controller | skipping: [controller]
2026-03-14 09:08:45.273366 | controller |
2026-03-14 09:08:45.273401 | controller | TASK [Deploy OpenShift cluster using dev-scripts name=devscripts] **************
2026-03-14 09:08:45.273409 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.027)       0:02:47.996 ********
2026-03-14 09:08:45.273416 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.027)       0:02:47.994 ********
2026-03-14 09:08:45.273428 | controller | skipping: [controller]
2026-03-14 09:08:45.480559 | controller |
2026-03-14 09:08:45.480592 | controller | TASK [openshift_login : Ensure output directory exists path={{ cifmw_openshift_login_basedir }}/artifacts, state=directory, mode=0755] ***
2026-03-14 09:08:45.480599 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.025)       0:02:48.021 ********
2026-03-14 09:08:45.480604 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.025)       0:02:48.020 ********
2026-03-14 09:08:45.480613 | controller | ok: [controller]
2026-03-14 09:08:45.520835 | controller |
2026-03-14 09:08:45.520867 | controller | TASK [openshift_login : OpenShift login _raw_params=login.yml] *****************
2026-03-14 09:08:45.520874 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.207)       0:02:48.228 ********
2026-03-14 09:08:45.520878 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.207)       0:02:48.227 ********
2026-03-14 09:08:45.520887 | controller | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/openshift_login/tasks/login.yml for controller
2026-03-14 09:08:45.558177 | controller |
2026-03-14 09:08:45.558210 | controller | TASK [openshift_login : Check if the password file is present path={{ cifmw_openshift_login_password_file | default(cifmw_openshift_password_file) }}] ***
2026-03-14 09:08:45.558217 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.040)       0:02:48.269 ********
2026-03-14 09:08:45.558222 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.040)       0:02:48.267 ********
2026-03-14 09:08:45.558231 | controller | skipping: [controller]
2026-03-14 09:08:45.594275 | controller |
2026-03-14 09:08:45.594323 | controller | TASK [openshift_login : Fetch user password content src={{ cifmw_openshift_login_password_file | default(cifmw_openshift_password_file) }}] ***
2026-03-14 09:08:45.594330 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.037)       0:02:48.306 ********
2026-03-14 09:08:45.594335 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.037)       0:02:48.305 ********
2026-03-14 09:08:45.594344 | controller | skipping: [controller]
2026-03-14 09:08:45.620948 | controller |
2026-03-14 09:08:45.620978 | controller | TASK [openshift_login : Set user password as a fact cifmw_openshift_login_password={{ cifmw_openshift_login_password_file_slurp.content | b64decode }}, cacheable=True] ***
2026-03-14 09:08:45.620985 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.036)       0:02:48.342 ********
2026-03-14 09:08:45.620990 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.036)       0:02:48.341 ********
2026-03-14 09:08:45.620998 | controller | skipping: [controller]
2026-03-14 09:08:45.661516 | controller |
2026-03-14 09:08:45.661548 | controller | TASK [openshift_login : Set role variables cifmw_openshift_login_kubeconfig={{
2026-03-14 09:08:45.661555 | controller |  cifmw_openshift_login_kubeconfig |
2026-03-14 09:08:45.661559 | controller |  default(cifmw_openshift_kubeconfig) |
2026-03-14 09:08:45.661564 | controller |  default(
2026-03-14 09:08:45.661569 | controller |    ansible_env.KUBECONFIG if 'KUBECONFIG' in ansible_env else
2026-03-14 09:08:45.661573 | controller |    cifmw_openshift_login_kubeconfig_default_path
2026-03-14 09:08:45.661578 | controller |  ) | trim
2026-03-14 09:08:45.661583 | controller | }}, cifmw_openshift_login_user={{ cifmw_openshift_login_user | default(cifmw_openshift_user) | default(omit) }}, cifmw_openshift_login_password={{ cifmw_openshift_login_password | default(cifmw_openshift_password) | default(omit) }}, cifmw_openshift_login_api={{ cifmw_openshift_login_api | default(cifmw_openshift_api) | default(omit) }}, cifmw_openshift_login_cert_login={{ cifmw_openshift_login_cert_login | default(false)}}, cifmw_openshift_login_provided_token={{ cifmw_openshift_provided_token | default(omit) }}, cacheable=True] ***
2026-03-14 09:08:45.661589 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.026)       0:02:48.369 ********
2026-03-14 09:08:45.661593 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.026)       0:02:48.367 ********
2026-03-14 09:08:45.661602 | controller | ok: [controller]
2026-03-14 09:08:45.869878 | controller |
2026-03-14 09:08:45.869963 | controller | TASK [openshift_login : Check if kubeconfig exists path={{ cifmw_openshift_login_kubeconfig }}] ***
2026-03-14 09:08:45.869977 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.040)       0:02:48.409 ********
2026-03-14 09:08:45.869987 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.040)       0:02:48.408 ********
2026-03-14 09:08:45.870011 | controller | ok: [controller]
2026-03-14 09:08:45.913408 | controller |
2026-03-14 09:08:45.913466 | controller | TASK [openshift_login : Assert that enough data is provided to log in to OpenShift that=cifmw_openshift_login_kubeconfig_stat.stat.exists or (cifmw_openshift_login_provided_token is defined and cifmw_openshift_login_provided_token != '') or (
2026-03-14 09:08:45.913475 | controller |   (cifmw_openshift_login_user is defined) and
2026-03-14 09:08:45.913482 | controller |   (cifmw_openshift_login_password is defined) and
2026-03-14 09:08:45.913488 | controller |   (cifmw_openshift_login_api is defined)
2026-03-14 09:08:45.913495 | controller | ), msg=If an existing kubeconfig is not provided user/pwd or provided/initial token and API URL must be given] ***
2026-03-14 09:08:45.913501 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.208)       0:02:48.617 ********
2026-03-14 09:08:45.913508 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.208)       0:02:48.616 ********
2026-03-14 09:08:45.913529 | controller | ok: [controller] =>
2026-03-14 09:08:45.938373 | controller |     changed: false
2026-03-14 09:08:45.938429 | controller |     msg: All assertions passed
2026-03-14 09:08:45.938437 | controller |
2026-03-14 09:08:45.938444 | controller | TASK [openshift_login : Fetch kubeconfig content src={{ cifmw_openshift_login_kubeconfig }}] ***
2026-03-14 09:08:45.938452 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.043)       0:02:48.661 ********
2026-03-14 09:08:45.938463 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.043)       0:02:48.660 ********
2026-03-14 09:08:45.938484 | controller | skipping: [controller]
2026-03-14 09:08:45.966960 | controller |
2026-03-14 09:08:45.966999 | controller | TASK [openshift_login : Fetch x509 key based users cifmw_openshift_login_key_based_users={{
2026-03-14 09:08:45.967008 | controller |   (
2026-03-14 09:08:45.967015 | controller |     cifmw_openshift_login_kubeconfig_content_b64.content |
2026-03-14 09:08:45.967021 | controller |     b64decode |
2026-03-14 09:08:45.967027 | controller |     from_yaml
2026-03-14 09:08:45.967033 | controller |   ). users | default([]) |
2026-03-14 09:08:45.967044 | controller |   selectattr('user.client-certificate-data', 'defined') |
2026-03-14 09:08:45.967050 | controller |   map(attribute="name") |
2026-03-14 09:08:45.967057 | controller |   map("split", "/") |
2026-03-14 09:08:45.967064 | controller |   map("first")
2026-03-14 09:08:45.967070 | controller | }}, cacheable=True] ***
2026-03-14 09:08:45.967076 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.025)       0:02:48.686 ********
2026-03-14 09:08:45.967082 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.025)       0:02:48.685 ********
2026-03-14 09:08:45.967094 | controller | skipping: [controller]
2026-03-14 09:08:45.998751 | controller |
2026-03-14 09:08:45.998784 | controller | TASK [openshift_login : Assign key based user if not provided and available cifmw_openshift_login_user={{
2026-03-14 09:08:45.998794 | controller |   (cifmw_openshift_login_assume_cert_system_user | ternary('system:', '')) +
2026-03-14 09:08:45.998801 | controller |   (cifmw_openshift_login_key_based_users | map('replace', 'system:', '') | unique | first)
2026-03-14 09:08:45.998808 | controller | }}, cifmw_openshift_login_cert_login=True, cacheable=True] ***
2026-03-14 09:08:45.998814 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.028)       0:02:48.714 ********
2026-03-14 09:08:45.998821 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.028)       0:02:48.713 ********
2026-03-14 09:08:45.998831 | controller | skipping: [controller]
2026-03-14 09:08:46.035251 | controller |
2026-03-14 09:08:46.035312 | controller | TASK [openshift_login : Set the retry count cifmw_openshift_login_retries_cnt={{
2026-03-14 09:08:46.035323 | controller |   0 if cifmw_openshift_login_retries_cnt is undefined else
2026-03-14 09:08:46.035329 | controller |   cifmw_openshift_login_retries_cnt|int + 1
2026-03-14 09:08:46.035336 | controller | }}] ***
2026-03-14 09:08:46.035342 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.031)       0:02:48.746 ********
2026-03-14 09:08:46.035348 | controller | Saturday 14 March 2026  09:08:45 +0000 (0:00:00.031)       0:02:48.745 ********
2026-03-14 09:08:46.035360 | controller | ok: [controller]
2026-03-14 09:08:46.073469 | controller |
2026-03-14 09:08:46.073630 | controller | TASK [openshift_login : Fetch token _raw_params=try_login.yml] *****************
2026-03-14 09:08:46.073641 | controller | Saturday 14 March 2026  09:08:46 +0000 (0:00:00.036)       0:02:48.783 ********
2026-03-14 09:08:46.073650 | controller | Saturday 14 March 2026  09:08:46 +0000 (0:00:00.036)       0:02:48.781 ********
2026-03-14 09:08:46.073667 | controller | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/openshift_login/tasks/try_login.yml for controller
2026-03-14 09:08:46.095419 | controller |
2026-03-14 09:08:46.095474 | controller | TASK [openshift_login : Try get OpenShift access token _raw_params=oc whoami -t] ***
2026-03-14 09:08:46.095483 | controller | Saturday 14 March 2026  09:08:46 +0000 (0:00:00.038)       0:02:48.821 ********
2026-03-14 09:08:46.095489 | controller | Saturday 14 March 2026  09:08:46 +0000 (0:00:00.038)       0:02:48.820 ********
2026-03-14 09:08:46.095503 | controller | skipping: [controller]
2026-03-14 09:08:46.730012 | controller |
2026-03-14 09:08:46.730073 | controller | TASK [openshift_login : Fetch OpenShift token output_dir={{ cifmw_openshift_login_basedir }}/artifacts, script=oc login {%- if cifmw_openshift_login_provided_token is not defined %} {%- if cifmw_openshift_login_user is defined %} -u {{ cifmw_openshift_login_user }} {%- endif %} {%- if cifmw_openshift_login_password is defined %} -p {{ cifmw_openshift_login_password }} {%- endif %} {% else %} --token={{ cifmw_openshift_login_provided_token }} {%- endif %} {%- if cifmw_openshift_login_skip_tls_verify|bool %} --insecure-skip-tls-verify=true {%- endif %} {%- if cifmw_openshift_login_api is defined %} {{ cifmw_openshift_login_api }} {%- endif %}] ***
2026-03-14 09:08:46.730102 | controller | Saturday 14 March 2026  09:08:46 +0000 (0:00:00.021)       0:02:48.843 ********
2026-03-14 09:08:46.730119 | controller | Saturday 14 March 2026  09:08:46 +0000 (0:00:00.021)       0:02:48.842 ********
2026-03-14 09:08:46.730140 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_002_fetch_openshift.log
2026-03-14 09:08:46.771377 | controller | changed: [controller]
2026-03-14 09:08:46.771441 | controller |
2026-03-14 09:08:46.771460 | controller | TASK [openshift_login : Ensure kubeconfig is provided that=cifmw_openshift_login_kubeconfig != ""] ***
2026-03-14 09:08:46.771475 | controller | Saturday 14 March 2026  09:08:46 +0000 (0:00:00.633)       0:02:49.477 ********
2026-03-14 09:08:46.771488 | controller | Saturday 14 March 2026  09:08:46 +0000 (0:00:00.633)       0:02:49.476 ********
2026-03-14 09:08:46.771511 | controller | ok: [controller] =>
2026-03-14 09:08:47.084040 | controller |     changed: false
2026-03-14 09:08:47.084084 | controller |     msg: All assertions passed
2026-03-14 09:08:47.084090 | controller |
2026-03-14 09:08:47.084095 | controller | TASK [openshift_login : Fetch new OpenShift access token _raw_params=oc whoami -t] ***
2026-03-14 09:08:47.084100 | controller | Saturday 14 March 2026  09:08:46 +0000 (0:00:00.041)       0:02:49.518 ********
2026-03-14 09:08:47.084105 | controller | Saturday 14 March 2026  09:08:46 +0000 (0:00:00.041)       0:02:49.517 ********
2026-03-14 09:08:47.084114 | controller | changed: [controller]
2026-03-14 09:08:47.114680 | controller |
2026-03-14 09:08:47.114710 | controller | TASK [openshift_login : Set new OpenShift token cifmw_openshift_login_token={{
2026-03-14 09:08:47.114716 | controller |   (not cifmw_openshift_login_new_token_out.skipped | default(false)) |
2026-03-14 09:08:47.114721 | controller |   ternary(cifmw_openshift_login_new_token_out.stdout, cifmw_openshift_login_whoami_out.stdout)
2026-03-14 09:08:47.114726 | controller | }}, cacheable=True] ***
2026-03-14 09:08:47.114730 | controller | Saturday 14 March 2026  09:08:47 +0000 (0:00:00.313)       0:02:49.832 ********
2026-03-14 09:08:47.114735 | controller | Saturday 14 March 2026  09:08:47 +0000 (0:00:00.313)       0:02:49.830 ********
2026-03-14 09:08:47.114744 | controller | ok: [controller]
2026-03-14 09:08:47.443277 | controller |
2026-03-14 09:08:47.443343 | controller | TASK [openshift_login : Fetch OpenShift API URL _raw_params=oc whoami --show-server=true] ***
2026-03-14 09:08:47.443352 | controller | Saturday 14 March 2026  09:08:47 +0000 (0:00:00.030)       0:02:49.862 ********
2026-03-14 09:08:47.443359 | controller | Saturday 14 March 2026  09:08:47 +0000 (0:00:00.030)       0:02:49.861 ********
2026-03-14 09:08:47.443370 | controller | changed: [controller]
2026-03-14 09:08:47.742219 | controller |
2026-03-14 09:08:47.742259 | controller | TASK [openshift_login : Fetch OpenShift kubeconfig context _raw_params=oc whoami -c] ***
2026-03-14 09:08:47.742270 | controller | Saturday 14 March 2026  09:08:47 +0000 (0:00:00.328)       0:02:50.191 ********
2026-03-14 09:08:47.742281 | controller | Saturday 14 March 2026  09:08:47 +0000 (0:00:00.328)       0:02:50.190 ********
2026-03-14 09:08:47.742331 | controller | changed: [controller]
2026-03-14 09:08:48.051505 | controller |
2026-03-14 09:08:48.051536 | controller | TASK [openshift_login : Fetch OpenShift current user _raw_params=oc whoami] ****
2026-03-14 09:08:48.051543 | controller | Saturday 14 March 2026  09:08:47 +0000 (0:00:00.299)       0:02:50.490 ********
2026-03-14 09:08:48.051547 | controller | Saturday 14 March 2026  09:08:47 +0000 (0:00:00.299)       0:02:50.489 ********
2026-03-14 09:08:48.051556 | controller | changed: [controller]
2026-03-14 09:08:48.095152 | controller |
2026-03-14 09:08:48.095187 | controller | TASK [openshift_login : Set OpenShift user, context and API facts cifmw_openshift_login_api={{ cifmw_openshift_login_api_out.stdout }}, cifmw_openshift_login_context={{ cifmw_openshift_login_context_out.stdout }}, cifmw_openshift_login_user={{ _oauth_user }}, cifmw_openshift_kubeconfig={{ cifmw_openshift_login_kubeconfig }}, cifmw_openshift_api={{ cifmw_openshift_login_api_out.stdout }}, cifmw_openshift_context={{ cifmw_openshift_login_context_out.stdout }}, cifmw_openshift_user={{ _oauth_user }}, cifmw_openshift_token={{ cifmw_openshift_login_token | default(omit) }}, cifmw_install_yamls_environment={{  ( cifmw_install_yamls_environment |
2026-03-14 09:08:48.095199 | controller |       combine({'KUBECONFIG': cifmw_openshift_login_kubeconfig})
2026-03-14 09:08:48.095205 | controller |     ) if cifmw_install_yamls_environment is defined else omit
2026-03-14 09:08:48.095209 | controller | }}, cacheable=True] ***
2026-03-14 09:08:48.095214 | controller | Saturday 14 March 2026  09:08:48 +0000 (0:00:00.309)       0:02:50.799 ********
2026-03-14 09:08:48.095219 | controller | Saturday 14 March 2026  09:08:48 +0000 (0:00:00.309)       0:02:50.798 ********
2026-03-14 09:08:48.095228 | controller | ok: [controller]
2026-03-14 09:08:48.616745 | controller |
2026-03-14 09:08:48.616818 | controller | TASK [openshift_login : Create the openshift_login parameters file dest={{ cifmw_basedir }}/artifacts/parameters/openshift-login-params.yml, content={{ cifmw_openshift_login_params_content | from_yaml | to_nice_yaml }}, mode=0600] ***
2026-03-14 09:08:48.616839 | controller | Saturday 14 March 2026  09:08:48 +0000 (0:00:00.043)       0:02:50.843 ********
2026-03-14 09:08:48.616856 | controller | Saturday 14 March 2026  09:08:48 +0000 (0:00:00.043)       0:02:50.841 ********
2026-03-14 09:08:48.616883 | controller | changed: [controller]
2026-03-14 09:08:48.937638 | controller |
2026-03-14 09:08:48.937678 | controller | TASK [openshift_login : Read the install yamls parameters file path={{ cifmw_basedir }}/artifacts/parameters/install-yamls-params.yml] ***
2026-03-14 09:08:48.937688 | controller | Saturday 14 March 2026  09:08:48 +0000 (0:00:00.521)       0:02:51.364 ********
2026-03-14 09:08:48.937695 | controller | Saturday 14 March 2026  09:08:48 +0000 (0:00:00.521)       0:02:51.363 ********
2026-03-14 09:08:48.937708 | controller | ok: [controller]
2026-03-14 09:08:49.578131 | controller |
2026-03-14 09:08:49.578220 | controller | TASK [openshift_login : Append the KUBECONFIG to the install yamls parameters content={{
2026-03-14 09:08:49.578229 | controller |   cifmw_openshift_login_install_yamls_artifacts_slurp['content'] | b64decode | from_yaml |
2026-03-14 09:08:49.578236 | controller |   combine(
2026-03-14 09:08:49.578242 | controller |     {
2026-03-14 09:08:49.578249 | controller |       'cifmw_install_yamls_environment': {
2026-03-14 09:08:49.578256 | controller |         'KUBECONFIG': cifmw_openshift_login_kubeconfig
2026-03-14 09:08:49.578263 | controller |       }
2026-03-14 09:08:49.578270 | controller |     }, recursive=true) | to_nice_yaml
2026-03-14 09:08:49.578276 | controller | }}, dest={{ cifmw_basedir }}/artifacts/parameters/install-yamls-params.yml, mode=0600] ***
2026-03-14 09:08:49.578283 | controller | Saturday 14 March 2026  09:08:48 +0000 (0:00:00.321)       0:02:51.685 ********
2026-03-14 09:08:49.578316 | controller | Saturday 14 March 2026  09:08:48 +0000 (0:00:00.321)       0:02:51.684 ********
2026-03-14 09:08:49.578333 | controller | changed: [controller]
2026-03-14 09:08:49.779821 | controller |
2026-03-14 09:08:49.779854 | controller | TASK [openshift_setup : Ensure output directory exists path={{ cifmw_openshift_setup_basedir }}/artifacts, state=directory, mode=0755] ***
2026-03-14 09:08:49.779861 | controller | Saturday 14 March 2026  09:08:49 +0000 (0:00:00.640)       0:02:52.326 ********
2026-03-14 09:08:49.779866 | controller | Saturday 14 March 2026  09:08:49 +0000 (0:00:00.640)       0:02:52.324 ********
2026-03-14 09:08:49.779875 | controller | ok: [controller]
2026-03-14 09:08:49.822729 | controller |
2026-03-14 09:08:49.822795 | controller | TASK [openshift_setup : Fetch namespaces to create cifmw_openshift_setup_namespaces={{
2026-03-14 09:08:49.822802 | controller |   ((
2026-03-14 09:08:49.822807 | controller |       ([cifmw_install_yamls_defaults['NAMESPACE']] +
2026-03-14 09:08:49.822812 | controller |         ([cifmw_install_yamls_defaults['OPERATOR_NAMESPACE']] if 'OPERATOR_NAMESPACE' is in cifmw_install_yamls_defaults else [])
2026-03-14 09:08:49.822817 | controller |       ) if cifmw_install_yamls_defaults is defined else []
2026-03-14 09:08:49.822821 | controller |   ) + cifmw_openshift_setup_create_namespaces) | unique
2026-03-14 09:08:49.822825 | controller | }}] ***
2026-03-14 09:08:49.822830 | controller | Saturday 14 March 2026  09:08:49 +0000 (0:00:00.201)       0:02:52.527 ********
2026-03-14 09:08:49.822841 | controller | Saturday 14 March 2026  09:08:49 +0000 (0:00:00.201)       0:02:52.526 ********
2026-03-14 09:08:49.822855 | controller | ok: [controller]
2026-03-14 09:08:51.603489 | controller |
2026-03-14 09:08:51.603519 | controller | TASK [openshift_setup : Create required namespaces kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit) }}, name={{ item }}, kind=Namespace, state=present] ***
2026-03-14 09:08:51.603527 | controller | Saturday 14 March 2026  09:08:49 +0000 (0:00:00.042)       0:02:52.570 ********
2026-03-14 09:08:51.603532 | controller | Saturday 14 March 2026  09:08:49 +0000 (0:00:00.042)       0:02:52.569 ********
2026-03-14 09:08:51.603542 | controller | changed: [controller] => (item=openstack)
2026-03-14 09:08:51.623262 | controller | changed: [controller] => (item=openstack-operators)
2026-03-14 09:08:51.623335 | controller |
2026-03-14 09:08:51.623343 | controller | TASK [openshift_setup : Get internal OpenShift registry route kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, kind=Route, name=default-route, namespace=openshift-image-registry] ***
2026-03-14 09:08:51.623349 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:01.781)       0:02:54.351 ********
2026-03-14 09:08:51.623355 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:01.781)       0:02:54.350 ********
2026-03-14 09:08:51.623367 | controller | skipping: [controller]
2026-03-14 09:08:51.647585 | controller |
2026-03-14 09:08:51.647620 | controller | TASK [openshift_setup : Allow anonymous image-pulls in CRC registry for targeted namespaces state=present, kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, definition={'kind': 'RoleBinding', 'apiVersion': 'rbac.authorization.k8s.io/v1', 'metadata': {'name': 'system:image-puller', 'namespace': '{{ item }}'}, 'subjects': [{'kind': 'User', 'name': 'system:anonymous'}, {'kind': 'User', 'name': 'system:unauthenticated'}], 'roleRef': {'kind': 'ClusterRole', 'name': 'system:image-puller'}}] ***
2026-03-14 09:08:51.647631 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.019)       0:02:54.371 ********
2026-03-14 09:08:51.647638 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.019)       0:02:54.370 ********
2026-03-14 09:08:51.647648 | controller | skipping: [controller] => (item=openstack)
2026-03-14 09:08:51.672000 | controller | skipping: [controller] => (item=openstack-operators)
2026-03-14 09:08:51.672033 | controller | skipping: [controller]
2026-03-14 09:08:51.672039 | controller |
2026-03-14 09:08:51.672045 | controller | TASK [openshift_setup : Wait for the image registry to be ready kind=Deployment, name=image-registry, namespace=openshift-image-registry, kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, wait=True, wait_sleep=10, wait_timeout=600, wait_condition={'type': 'Available', 'status': 'True'}] ***
2026-03-14 09:08:51.672051 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.024)       0:02:54.395 ********
2026-03-14 09:08:51.672055 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.024)       0:02:54.394 ********
2026-03-14 09:08:51.672065 | controller | skipping: [controller]
2026-03-14 09:08:51.696269 | controller |
2026-03-14 09:08:51.696324 | controller | TASK [openshift_setup : Login into OpenShift internal registry output_dir={{ cifmw_openshift_setup_basedir }}/artifacts, script=podman login -u {{ cifmw_openshift_user }} -p {{ cifmw_openshift_token }} {%- if cifmw_openshift_setup_skip_internal_registry_tls_verify|bool %} --tls-verify=false {%- endif %} {{ cifmw_openshift_setup_registry_default_route.resources[0].spec.host }}] ***
2026-03-14 09:08:51.696331 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.024)       0:02:54.420 ********
2026-03-14 09:08:51.696336 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.024)       0:02:54.418 ********
2026-03-14 09:08:51.696346 | controller | skipping: [controller]
2026-03-14 09:08:51.720791 | controller |
2026-03-14 09:08:51.720829 | controller | TASK [Ensure we have custom CA installed on host role=install_ca] **************
2026-03-14 09:08:51.720846 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.024)       0:02:54.444 ********
2026-03-14 09:08:51.720852 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.024)       0:02:54.443 ********
2026-03-14 09:08:51.720863 | controller | skipping: [controller]
2026-03-14 09:08:51.742968 | controller |
2026-03-14 09:08:51.743000 | controller | TASK [openshift_setup : Update ca bundle _raw_params=update-ca-trust extract] ***
2026-03-14 09:08:51.743006 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.024)       0:02:54.468 ********
2026-03-14 09:08:51.743011 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.024)       0:02:54.467 ********
2026-03-14 09:08:51.743019 | controller | skipping: [controller]
2026-03-14 09:08:51.765024 | controller |
2026-03-14 09:08:51.765055 | controller | TASK [openshift_setup : Slurp CAs file src={{ cifmw_openshift_setup_ca_bundle_path }}] ***
2026-03-14 09:08:51.765062 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.022)       0:02:54.491 ********
2026-03-14 09:08:51.765066 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.022)       0:02:54.489 ********
2026-03-14 09:08:51.765075 | controller | skipping: [controller]
2026-03-14 09:08:51.785578 | controller |
2026-03-14 09:08:51.785608 | controller | TASK [openshift_setup : Create config map with registry CAs kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, definition={'apiVersion': 'v1', 'kind': 'ConfigMap', 'metadata': {'namespace': 'openshift-config', 'name': 'registry-cas'}, 'data': '{{ _config_map_data | items2dict  }}'}] ***
2026-03-14 09:08:51.785615 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.022)       0:02:54.513 ********
2026-03-14 09:08:51.785620 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.022)       0:02:54.511 ********
2026-03-14 09:08:51.785627 | controller | skipping: [controller]
2026-03-14 09:08:51.807465 | controller |
2026-03-14 09:08:51.807490 | controller | TASK [openshift_setup : Install Red Hat CA for pulling images from internal registry kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, merge_type=merge, definition={'apiVersion': 'config.openshift.io/v1', 'kind': 'Image', 'metadata': {'name': 'cluster'}, 'spec': {'additionalTrustedCA': {'name': 'registry-cas'}}}] ***
2026-03-14 09:08:51.807499 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.020)       0:02:54.533 ********
2026-03-14 09:08:51.807504 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.020)       0:02:54.532 ********
2026-03-14 09:08:51.807512 | controller | skipping: [controller]
2026-03-14 09:08:51.829299 | controller |
2026-03-14 09:08:51.829325 | controller | TASK [openshift_setup : Add insecure registry kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, merge_type=merge, definition={'apiVersion': 'config.openshift.io/v1', 'kind': 'Image', 'metadata': {'name': 'cluster'}, 'spec': {'registrySources': {'insecureRegistries': ['{{ cifmw_update_containers_registry }}'], 'allowedRegistries': '{{ all_registries }}'}}}] ***
2026-03-14 09:08:51.829333 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.021)       0:02:54.555 ********
2026-03-14 09:08:51.829339 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.021)       0:02:54.554 ********
2026-03-14 09:08:51.829346 | controller | skipping: [controller]
2026-03-14 09:08:51.854679 | controller |
2026-03-14 09:08:51.854717 | controller | TASK [openshift_setup : Create a ICSP with repository digest mirrors kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, definition={'apiVersion': 'operator.openshift.io/v1alpha1', 'kind': 'ImageContentSourcePolicy', 'metadata': {'name': 'registry-digest-mirrors'}, 'spec': {'repositoryDigestMirrors': '{{ cifmw_openshift_setup_digest_mirrors }}'}}] ***
2026-03-14 09:08:51.854724 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.021)       0:02:54.577 ********
2026-03-14 09:08:51.854736 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.021)       0:02:54.576 ********
2026-03-14 09:08:51.854745 | controller | skipping: [controller]
2026-03-14 09:08:52.797058 | controller |
2026-03-14 09:08:52.797090 | controller | TASK [openshift_setup : Gather network.operator info kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, api_version=operator.openshift.io/v1, kind=Network, name=cluster] ***
2026-03-14 09:08:52.797097 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.025)       0:02:54.602 ********
2026-03-14 09:08:52.797102 | controller | Saturday 14 March 2026  09:08:51 +0000 (0:00:00.025)       0:02:54.601 ********
2026-03-14 09:08:52.797112 | controller | ok: [controller]
2026-03-14 09:08:53.687239 | controller |
2026-03-14 09:08:53.687303 | controller | TASK [openshift_setup : Patch network operator api_version=operator.openshift.io/v1, kubeconfig={{ cifmw_openshift_kubeconfig }}, kind=Network, name=cluster, persist_config=True, patch=[{'path': '/spec/defaultNetwork/ovnKubernetesConfig/gatewayConfig/routingViaHost', 'value': True, 'op': 'replace'}, {'path': '/spec/defaultNetwork/ovnKubernetesConfig/gatewayConfig/ipForwarding', 'value': 'Global', 'op': 'replace'}]] ***
2026-03-14 09:08:53.687317 | controller | Saturday 14 March 2026  09:08:52 +0000 (0:00:00.942)       0:02:55.545 ********
2026-03-14 09:08:53.687324 | controller | Saturday 14 March 2026  09:08:52 +0000 (0:00:00.942)       0:02:55.543 ********
2026-03-14 09:08:53.687337 | controller | changed: [controller]
2026-03-14 09:08:54.664006 | controller |
2026-03-14 09:08:54.664098 | controller | TASK [openshift_setup : Patch samples registry configuration kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, api_version=samples.operator.openshift.io/v1, kind=Config, name=cluster, patch=[{'op': 'replace', 'path': '/spec/samplesRegistry', 'value': '{{ cifmw_openshift_setup_samples_registry }}'}]] ***
2026-03-14 09:08:54.664113 | controller | Saturday 14 March 2026  09:08:53 +0000 (0:00:00.890)       0:02:56.435 ********
2026-03-14 09:08:54.664125 | controller | Saturday 14 March 2026  09:08:53 +0000 (0:00:00.890)       0:02:56.434 ********
2026-03-14 09:08:54.664145 | controller | changed: [controller]
2026-03-14 09:08:54.701260 | controller |
2026-03-14 09:08:54.701330 | controller | TASK [openshift_setup : Delete the pods from openshift-marketplace namespace kind=Pod, state=absent, delete_all=True, kubeconfig={{ cifmw_openshift_kubeconfig }}, namespace=openshift-marketplace] ***
2026-03-14 09:08:54.701341 | controller | Saturday 14 March 2026  09:08:54 +0000 (0:00:00.976)       0:02:57.411 ********
2026-03-14 09:08:54.701348 | controller | Saturday 14 March 2026  09:08:54 +0000 (0:00:00.976)       0:02:57.410 ********
2026-03-14 09:08:54.701361 | controller | skipping: [controller]
2026-03-14 09:08:54.734138 | controller |
2026-03-14 09:08:54.734182 | controller | TASK [openshift_setup : Wait for openshift-marketplace pods to be running _raw_params=oc wait pod --all --for=condition=Ready -n openshift-marketplace --timeout=1m] ***
2026-03-14 09:08:54.734192 | controller | Saturday 14 March 2026  09:08:54 +0000 (0:00:00.037)       0:02:57.449 ********
2026-03-14 09:08:54.734199 | controller | Saturday 14 March 2026  09:08:54 +0000 (0:00:00.037)       0:02:57.447 ********
2026-03-14 09:08:54.734211 | controller | skipping: [controller]
2026-03-14 09:08:54.822758 | controller |
2026-03-14 09:08:54.822790 | controller | TASK [Deploy Observability operator. name=openshift_obs] ***********************
2026-03-14 09:08:54.822796 | controller | Saturday 14 March 2026  09:08:54 +0000 (0:00:00.032)       0:02:57.481 ********
2026-03-14 09:08:54.822801 | controller | Saturday 14 March 2026  09:08:54 +0000 (0:00:00.032)       0:02:57.480 ********
2026-03-14 09:08:54.822810 | controller |
2026-03-14 09:08:56.309253 | controller | TASK [openshift_obs : Install cluster observability operator. definition={{cifmw_openshift_obs_definition }}, kubeconfig={{ cifmw_openshift_kubeconfig }}, state=present] ***
2026-03-14 09:08:56.309321 | controller | Saturday 14 March 2026  09:08:54 +0000 (0:00:00.088)       0:02:57.570 ********
2026-03-14 09:08:56.309332 | controller | Saturday 14 March 2026  09:08:54 +0000 (0:00:00.088)       0:02:57.569 ********
2026-03-14 09:08:56.309354 | controller | changed: [controller]
2026-03-14 09:09:48.191254 | controller |
2026-03-14 09:09:48.191335 | controller | TASK [openshift_obs : Wait for observability operator deployment kind=Deployment, namespace=openshift-operators, name=observability-operator, wait=True, wait_timeout=300, wait_condition={'type': 'Available', 'status': 'True'}, kubeconfig={{ cifmw_openshift_kubeconfig }}] ***
2026-03-14 09:09:48.191347 | controller | Saturday 14 March 2026  09:08:56 +0000 (0:00:01.485)       0:02:59.056 ********
2026-03-14 09:09:48.191354 | controller | Saturday 14 March 2026  09:08:56 +0000 (0:00:01.485)       0:02:59.055 ********
2026-03-14 09:09:48.191366 | controller | ok: [controller]
2026-03-14 09:09:48.914025 | controller |
2026-03-14 09:09:48.914073 | controller | TASK [openshift_obs : Wait for observability-operator pod kind=Pod, namespace=openshift-operators, label_selectors=['app.kubernetes.io/name = observability-operator'], wait=True, wait_timeout=300, wait_condition={'type': 'Ready', 'status': 'True'}, kubeconfig={{ cifmw_openshift_kubeconfig }}] ***
2026-03-14 09:09:48.914080 | controller | Saturday 14 March 2026  09:09:48 +0000 (0:00:51.882)       0:03:50.939 ********
2026-03-14 09:09:48.914085 | controller | Saturday 14 March 2026  09:09:48 +0000 (0:00:51.882)       0:03:50.938 ********
2026-03-14 09:09:48.914095 | controller | ok: [controller]
2026-03-14 09:09:48.935910 | controller |
2026-03-14 09:09:48.935940 | controller | TASK [Deploy Metal3 BMHs name=deploy_bmh] **************************************
2026-03-14 09:09:48.935947 | controller | Saturday 14 March 2026  09:09:48 +0000 (0:00:00.722)       0:03:51.662 ********
2026-03-14 09:09:48.935951 | controller | Saturday 14 March 2026  09:09:48 +0000 (0:00:00.722)       0:03:51.660 ********
2026-03-14 09:09:48.935959 | controller | skipping: [controller]
2026-03-14 09:09:48.958589 | controller |
2026-03-14 09:09:48.958638 | controller | TASK [Install certmanager operator role name=cert_manager] *********************
2026-03-14 09:09:48.958645 | controller | Saturday 14 March 2026  09:09:48 +0000 (0:00:00.021)       0:03:51.684 ********
2026-03-14 09:09:48.958649 | controller | Saturday 14 March 2026  09:09:48 +0000 (0:00:00.021)       0:03:51.682 ********
2026-03-14 09:09:48.958660 | controller | skipping: [controller]
2026-03-14 09:09:48.982351 | controller |
2026-03-14 09:09:48.982396 | controller | TASK [Configure hosts networking using nmstate name=ci_nmstate] ****************
2026-03-14 09:09:48.982403 | controller | Saturday 14 March 2026  09:09:48 +0000 (0:00:00.022)       0:03:51.706 ********
2026-03-14 09:09:48.982407 | controller | Saturday 14 March 2026  09:09:48 +0000 (0:00:00.022)       0:03:51.705 ********
2026-03-14 09:09:48.982417 | controller | skipping: [controller]
2026-03-14 09:09:49.002741 | controller |
2026-03-14 09:09:49.002786 | controller | TASK [Configure multus networks name=ci_multus] ********************************
2026-03-14 09:09:49.002792 | controller | Saturday 14 March 2026  09:09:48 +0000 (0:00:00.023)       0:03:51.730 ********
2026-03-14 09:09:49.002797 | controller | Saturday 14 March 2026  09:09:48 +0000 (0:00:00.023)       0:03:51.729 ********
2026-03-14 09:09:49.002806 | controller | skipping: [controller]
2026-03-14 09:09:49.021626 | controller |
2026-03-14 09:09:49.021674 | controller | TASK [Deploy Sushy Emulator service pod name=sushy_emulator] *******************
2026-03-14 09:09:49.021680 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.020)       0:03:51.750 ********
2026-03-14 09:09:49.021685 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.020)       0:03:51.749 ********
2026-03-14 09:09:49.021694 | controller | skipping: [controller]
2026-03-14 09:09:49.040629 | controller |
2026-03-14 09:09:49.040675 | controller | TASK [Setup Libvirt on controller name=libvirt_manager] ************************
2026-03-14 09:09:49.040682 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.018)       0:03:51.769 ********
2026-03-14 09:09:49.040686 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.018)       0:03:51.768 ********
2026-03-14 09:09:49.040697 | controller | skipping: [controller]
2026-03-14 09:09:49.062966 | controller |
2026-03-14 09:09:49.063013 | controller | TASK [Prepare container package builder name=pkg_build] ************************
2026-03-14 09:09:49.063020 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.019)       0:03:51.788 ********
2026-03-14 09:09:49.063024 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.019)       0:03:51.787 ********
2026-03-14 09:09:49.063040 | controller | skipping: [controller]
2026-03-14 09:09:49.116702 | controller |
2026-03-14 09:09:49.116752 | controller | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] ***
2026-03-14 09:09:49.116759 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.022)       0:03:51.811 ********
2026-03-14 09:09:49.116765 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.022)       0:03:51.809 ********
2026-03-14 09:09:49.116775 | controller | ok: [controller]
2026-03-14 09:09:49.214197 | controller |
2026-03-14 09:09:49.214247 | controller | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] ***
2026-03-14 09:09:49.214253 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.053)       0:03:51.864 ********
2026-03-14 09:09:49.214258 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.053)       0:03:51.863 ********
2026-03-14 09:09:49.214269 | controller | ok: [controller]
2026-03-14 09:09:49.319897 | controller |
2026-03-14 09:09:49.319947 | controller | TASK [run_hook : Loop on hooks for post_infra _raw_params={{ hook.type }}.yml] ***
2026-03-14 09:09:49.319954 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.097)       0:03:51.962 ********
2026-03-14 09:09:49.319963 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.097)       0:03:51.961 ********
2026-03-14 09:09:49.319974 | controller | skipping: [controller]
2026-03-14 09:09:49.376656 | controller |
2026-03-14 09:09:49.376709 | controller | TASK [Load parameters dir={{ item }}] ******************************************
2026-03-14 09:09:49.376716 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.105)       0:03:52.067 ********
2026-03-14 09:09:49.376720 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.105)       0:03:52.066 ********
2026-03-14 09:09:49.376732 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts/parameters)
2026-03-14 09:09:49.407171 | controller | ok: [controller] => (item=/etc/ci/env)
2026-03-14 09:09:49.407223 | controller |
2026-03-14 09:09:49.407232 | controller | TASK [Ensure that the isolated net was configured for crc that=['crc_ci_bootstrap_networks_out is defined', 'crc_ci_bootstrap_networks_out[_crc_hostname] is defined', "crc_ci_bootstrap_networks_out[_crc_hostname]['default'] is defined"]] ***
2026-03-14 09:09:49.407239 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.056)       0:03:52.124 ********
2026-03-14 09:09:49.407245 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.056)       0:03:52.123 ********
2026-03-14 09:09:49.407255 | controller | ok: [controller] =>
2026-03-14 09:09:49.506485 | controller |     changed: false
2026-03-14 09:09:49.506548 | controller |     msg: All assertions passed
2026-03-14 09:09:49.506554 | controller |
2026-03-14 09:09:49.506560 | controller | TASK [Set facts for further usage within the framework cifmw_edpm_prepare_extra_vars={'NNCP_INTERFACE': '{{ crc_ci_bootstrap_networks_out.crc.default.iface }}', 'NETWORK_MTU': '{{ crc_ci_bootstrap_networks_out.crc.default.mtu }}', 'NNCP_DNS_SERVER': "{{\n  cifmw_nncp_dns_server |\n  default(crc_ci_bootstrap_networks_out[_crc_hostname].default.ip) |\n  split('/') | first\n}}"}] ***
2026-03-14 09:09:49.506565 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.030)       0:03:52.155 ********
2026-03-14 09:09:49.506570 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.030)       0:03:52.153 ********
2026-03-14 09:09:49.506584 | controller | ok: [controller]
2026-03-14 09:09:49.549034 | controller |
2026-03-14 09:09:49.549117 | controller | PLAY [Deploy Openstack Operators] **********************************************
2026-03-14 09:09:49.549128 | controller |
2026-03-14 09:09:49.549134 | controller | TASK [Use the locally built operators if any _local_operators_indexes={{
2026-03-14 09:09:49.549139 | controller |   _local_operators_indexes|default({}) |
2026-03-14 09:09:49.549145 | controller |   combine({ item.key.split('-')[0]|upper+'_IMG':
2026-03-14 09:09:49.549149 | controller |   cifmw_operator_build_output['operators'][item.key].image_catalog})
2026-03-14 09:09:49.549154 | controller | }}] ***
2026-03-14 09:09:49.549159 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.099)       0:03:52.254 ********
2026-03-14 09:09:49.549168 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.099)       0:03:52.253 ********
2026-03-14 09:09:49.549180 | controller | ok: [controller] => (item={'key': 'openstack-operator', 'value': {'git_commit_hash': '9a24db8c31721f3ef0623b08a5d9140ff7ecb702', 'git_src_dir': '/home/zuul-worker/src/github.com/openstack-k8s-operators/openstack-operator', 'image': '38.102.83.153:5001/openstack-k8s-operators/openstack-operator:9a24db8c31721f3ef0623b08a5d9140ff7ecb702', 'image_bundle': '38.102.83.153:5001/openstack-k8s-operators/openstack-operator-bundle:9a24db8c31721f3ef0623b08a5d9140ff7ecb702', 'image_catalog': '38.102.83.153:5001/openstack-k8s-operators/openstack-operator-index:9a24db8c31721f3ef0623b08a5d9140ff7ecb702'}})
2026-03-14 09:09:49.574841 | controller | ok: [controller] => (item={'key': 'watcher-operator', 'value': {'git_commit_hash': '3fad4a9eb56718f26ce2ec186bb570f2695f01c3', 'git_src_dir': '/home/zuul-worker/src/github.com/openstack-k8s-operators/watcher-operator', 'image': '38.102.83.153:5001/openstack-k8s-operators/watcher-operator:3fad4a9eb56718f26ce2ec186bb570f2695f01c3', 'image_bundle': '38.102.83.153:5001/openstack-k8s-operators/watcher-operator-bundle:3fad4a9eb56718f26ce2ec186bb570f2695f01c3', 'image_catalog': '38.102.83.153:5001/openstack-k8s-operators/watcher-operator-index:3fad4a9eb56718f26ce2ec186bb570f2695f01c3'}})
2026-03-14 09:09:49.574890 | controller |
2026-03-14 09:09:49.574897 | controller | TASK [Set install_yamls Makefile environment variables cifmw_edpm_prepare_common_env={{
2026-03-14 09:09:49.574902 | controller |   cifmw_install_yamls_environment |
2026-03-14 09:09:49.574907 | controller |   combine({'PATH': cifmw_path}) |
2026-03-14 09:09:49.574912 | controller |   combine(cifmw_edpm_prepare_extra_vars | default({}))
2026-03-14 09:09:49.574917 | controller | }}, cifmw_edpm_prepare_operators_build_output={{ operators_build_output }}, cifmw_edpm_prepare_make_openstack_env={{ _local_operators_indexes | combine(_openstack_operator_images) }}] ***
2026-03-14 09:09:49.574922 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.042)       0:03:52.297 ********
2026-03-14 09:09:49.574927 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.042)       0:03:52.295 ********
2026-03-14 09:09:49.574936 | controller | ok: [controller]
2026-03-14 09:09:49.959914 | controller |
2026-03-14 09:09:49.959947 | controller | TASK [detect if openstack operator is installed _raw_params=oc get sub --ignore-not-found=true -n openstack-operators -o name openstack-operator] ***
2026-03-14 09:09:49.959954 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.025)       0:03:52.323 ********
2026-03-14 09:09:49.959959 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.025)       0:03:52.321 ********
2026-03-14 09:09:49.959968 | controller | changed: [controller]
2026-03-14 09:09:49.986944 | controller |
2026-03-14 09:09:49.986976 | controller | TASK [Install openstack operator and wait for the csv to succeed name=install_yamls_makes, tasks_from=make_openstack_init] ***
2026-03-14 09:09:49.986982 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.385)       0:03:52.708 ********
2026-03-14 09:09:49.986987 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.385)       0:03:52.706 ********
2026-03-14 09:09:49.986995 | controller |
2026-03-14 09:09:50.025501 | controller | TASK [install_yamls_makes : Debug make_openstack_init_env var=make_openstack_init_env] ***
2026-03-14 09:09:50.025538 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.026)       0:03:52.734 ********
2026-03-14 09:09:50.025545 | controller | Saturday 14 March 2026  09:09:49 +0000 (0:00:00.026)       0:03:52.733 ********
2026-03-14 09:09:50.025554 | controller | ok: [controller] =>
2026-03-14 09:09:50.052451 | controller |     make_openstack_init_env:
2026-03-14 09:09:50.052478 | controller |       BMO_SETUP: false
2026-03-14 09:09:50.052484 | controller |       CHECKOUT_FROM_OPENSTACK_REF: 'true'
2026-03-14 09:09:50.052490 | controller |       KUBECONFIG: /home/zuul/.crc/machines/crc/kubeconfig
2026-03-14 09:09:50.052495 | controller |       NETWORK_MTU: 1500
2026-03-14 09:09:50.052499 | controller |       NNCP_DNS_SERVER: 192.168.122.10
2026-03-14 09:09:50.052503 | controller |       NNCP_INTERFACE: ens7
2026-03-14 09:09:50.052508 | controller |       OPENSTACK_BUNDLE_IMG: 38.102.83.153:5001/openstack-k8s-operators/openstack-operator-bundle:9a24db8c31721f3ef0623b08a5d9140ff7ecb702
2026-03-14 09:09:50.052520 | controller |       OPENSTACK_IMG: 38.102.83.153:5001/openstack-k8s-operators/openstack-operator-index:9a24db8c31721f3ef0623b08a5d9140ff7ecb702
2026-03-14 09:09:50.052524 | controller |       OPENSTACK_K8S_BRANCH: 18.0-fr5
2026-03-14 09:09:50.052528 | controller |       OUT: /home/zuul/ci-framework-data/artifacts/manifests
2026-03-14 09:09:50.052533 | controller |       OUTPUT_DIR: /home/zuul/ci-framework-data/artifacts/edpm
2026-03-14 09:09:50.052537 | controller |       PATH: /home/zuul/.crc/bin:/home/zuul/.crc/bin/oc:/home/zuul/bin:/home/zuul/.local/bin:/home/zuul/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
2026-03-14 09:09:50.052542 | controller |       WATCHER_BRANCH: ''
2026-03-14 09:09:50.052547 | controller |       WATCHER_IMG: 38.102.83.153:5001/openstack-k8s-operators/watcher-operator-index:3fad4a9eb56718f26ce2ec186bb570f2695f01c3
2026-03-14 09:09:50.052551 | controller |       WATCHER_REPO: /home/zuul/src/github.com/openstack-k8s-operators/watcher-operator
2026-03-14 09:09:50.052556 | controller |
2026-03-14 09:09:50.052561 | controller | TASK [install_yamls_makes : Debug make_openstack_init_params var=make_openstack_init_params] ***
2026-03-14 09:09:50.052565 | controller | Saturday 14 March 2026  09:09:50 +0000 (0:00:00.038)       0:03:52.773 ********
2026-03-14 09:09:50.052570 | controller | Saturday 14 March 2026  09:09:50 +0000 (0:00:00.038)       0:03:52.772 ********
2026-03-14 09:09:50.052577 | controller | skipping: [controller]
2026-03-14 09:15:20.737396 | controller |
2026-03-14 09:15:20.737438 | controller | TASK [install_yamls_makes : Run openstack_init output_dir={{ cifmw_basedir }}/artifacts, chdir=/home/zuul/src/github.com/openstack-k8s-operators/install_yamls, script=make openstack_init, dry_run={{ make_openstack_init_dryrun|default(false)|bool }}, extra_args={{ dict((make_openstack_init_env|default({})), **(make_openstack_init_params|default({}))) }}] ***
2026-03-14 09:15:20.737445 | controller | Saturday 14 March 2026  09:09:50 +0000 (0:00:00.027)       0:03:52.800 ********
2026-03-14 09:15:20.737451 | controller | Saturday 14 March 2026  09:09:50 +0000 (0:00:00.027)       0:03:52.799 ********
2026-03-14 09:15:20.737459 | controller | [WARNING]: conditional statements should not include jinja2 templating
2026-03-14 09:15:20.752162 | controller | delimiters such as {{ }} or {% %}. Found: {{ make_openstack_init_until |
2026-03-14 09:15:20.752190 | controller | default(true) }}
2026-03-14 09:15:20.752200 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_003_run_openstack.log
2026-03-14 09:15:20.781413 | controller | changed: [controller]
2026-03-14 09:15:20.781448 | controller |
2026-03-14 09:15:20.781454 | controller | TASK [Run hooks after installing openstack name=run_hook] **********************
2026-03-14 09:15:20.781459 | controller | Saturday 14 March 2026  09:15:20 +0000 (0:05:30.699)       0:09:23.500 ********
2026-03-14 09:15:20.781463 | controller | Saturday 14 March 2026  09:15:20 +0000 (0:05:30.699)       0:09:23.499 ********
2026-03-14 09:15:20.781473 | controller |
2026-03-14 09:15:20.860138 | controller | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] ***
2026-03-14 09:15:20.860173 | controller | Saturday 14 March 2026  09:15:20 +0000 (0:00:00.029)       0:09:23.529 ********
2026-03-14 09:15:20.860181 | controller | Saturday 14 March 2026  09:15:20 +0000 (0:00:00.029)       0:09:23.528 ********
2026-03-14 09:15:20.860190 | controller | ok: [controller]
2026-03-14 09:15:20.964154 | controller |
2026-03-14 09:15:20.964195 | controller | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] ***
2026-03-14 09:15:20.964202 | controller | Saturday 14 March 2026  09:15:20 +0000 (0:00:00.078)       0:09:23.608 ********
2026-03-14 09:15:20.964207 | controller | Saturday 14 March 2026  09:15:20 +0000 (0:00:00.078)       0:09:23.606 ********
2026-03-14 09:15:20.964216 | controller | ok: [controller]
2026-03-14 09:15:21.115002 | controller |
2026-03-14 09:15:21.115042 | controller | TASK [run_hook : Loop on hooks for post_install_operators_kuttl_from_operator _raw_params={{ hook.type }}.yml] ***
2026-03-14 09:15:21.115062 | controller | Saturday 14 March 2026  09:15:20 +0000 (0:00:00.103)       0:09:23.711 ********
2026-03-14 09:15:21.115069 | controller | Saturday 14 March 2026  09:15:20 +0000 (0:00:00.103)       0:09:23.710 ********
2026-03-14 09:15:21.115083 | controller | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/run_hook/tasks/playbook.yml for controller => (item={'name': 'Deploy watcher operator', 'type': 'playbook', 'source': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator/ci/playbooks/deploy_watcher_operator.yaml', 'extra_vars': {'content_provider_os_registry_url': '38.102.83.153:5001/podified-master-centos10', 'watcher_catalog_image': '38.102.83.153:5001/openstack-k8s-operators/watcher-operator-index:3fad4a9eb56718f26ce2ec186bb570f2695f01c3', 'watcher_services_tag': 'watcher_latest', 'watcher_repo': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator'}})
2026-03-14 09:15:21.159864 | controller |
2026-03-14 09:15:21.159895 | controller | TASK [run_hook : Set playbook path for Deploy watcher operator cifmw_basedir={{ _bdir }}, hook_name={{ _hook_name }}, playbook_path={{ _play | realpath }}, log_path={{ _bdir }}/logs/{{ step }}_{{ _hook_name }}.log, extra_vars=-e namespace={{ cifmw_openstack_namespace }} {%- if hook.extra_vars is defined and hook.extra_vars|length > 0 -%} {% for key,value in hook.extra_vars.items() -%} {%- if key == 'file' %} -e "@{{ value }}" {%- else %} -e "{{ key }}={{ value }}" {%- endif %} {%- endfor %} {%- endif %}] ***
2026-03-14 09:15:21.159902 | controller | Saturday 14 March 2026  09:15:21 +0000 (0:00:00.150)       0:09:23.862 ********
2026-03-14 09:15:21.159908 | controller | Saturday 14 March 2026  09:15:21 +0000 (0:00:00.150)       0:09:23.861 ********
2026-03-14 09:15:21.159916 | controller | ok: [controller]
2026-03-14 09:15:21.372216 | controller |
2026-03-14 09:15:21.372251 | controller | TASK [run_hook : Get file stat path={{ playbook_path }}] ***********************
2026-03-14 09:15:21.372257 | controller | Saturday 14 March 2026  09:15:21 +0000 (0:00:00.045)       0:09:23.908 ********
2026-03-14 09:15:21.372261 | controller | Saturday 14 March 2026  09:15:21 +0000 (0:00:00.045)       0:09:23.906 ********
2026-03-14 09:15:21.372270 | controller | ok: [controller]
2026-03-14 09:15:21.394493 | controller |
2026-03-14 09:15:21.394527 | controller | TASK [run_hook : Fail if playbook doesn't exist msg=Playbook {{ playbook_path }} doesn't seem to exist.] ***
2026-03-14 09:15:21.394533 | controller | Saturday 14 March 2026  09:15:21 +0000 (0:00:00.212)       0:09:24.120 ********
2026-03-14 09:15:21.394538 | controller | Saturday 14 March 2026  09:15:21 +0000 (0:00:00.212)       0:09:24.118 ********
2026-03-14 09:15:21.394547 | controller | skipping: [controller]
2026-03-14 09:15:21.604389 | controller |
2026-03-14 09:15:21.604429 | controller | TASK [run_hook : Get parameters files paths={{
2026-03-14 09:15:21.604438 | controller |   (cifmw_basedir, 'artifacts/parameters') | path_join
2026-03-14 09:15:21.604445 | controller | }}, file_type=file, patterns=*.yml] ***
2026-03-14 09:15:21.604452 | controller | Saturday 14 March 2026  09:15:21 +0000 (0:00:00.022)       0:09:24.142 ********
2026-03-14 09:15:21.604459 | controller | Saturday 14 March 2026  09:15:21 +0000 (0:00:00.022)       0:09:24.141 ********
2026-03-14 09:15:21.604471 | controller | ok: [controller]
2026-03-14 09:15:21.633079 | controller |
2026-03-14 09:15:21.633118 | controller | TASK [run_hook : Add parameters artifacts as extra variables extra_vars={{ extra_vars }} {% for file in cifmw_run_hook_parameters_files.files %} -e "@{{ file.path }}" {%- endfor %}] ***
2026-03-14 09:15:21.633128 | controller | Saturday 14 March 2026  09:15:21 +0000 (0:00:00.209)       0:09:24.352 ********
2026-03-14 09:15:21.633135 | controller | Saturday 14 March 2026  09:15:21 +0000 (0:00:00.209)       0:09:24.351 ********
2026-03-14 09:15:21.633147 | controller | ok: [controller]
2026-03-14 09:15:21.859647 | controller |
2026-03-14 09:15:21.859692 | controller | TASK [run_hook : Ensure log directory exists path={{ log_path | dirname }}, state=directory, mode=0755] ***
2026-03-14 09:15:21.859701 | controller | Saturday 14 March 2026  09:15:21 +0000 (0:00:00.028)       0:09:24.380 ********
2026-03-14 09:15:21.859707 | controller | Saturday 14 March 2026  09:15:21 +0000 (0:00:00.028)       0:09:24.379 ********
2026-03-14 09:15:21.859720 | controller | ok: [controller]
2026-03-14 09:15:22.119317 | controller |
2026-03-14 09:15:22.119360 | controller | TASK [run_hook : Ensure artifacts directory exists path={{ cifmw_basedir }}/artifacts, state=directory, mode=0755] ***
2026-03-14 09:15:22.119370 | controller | Saturday 14 March 2026  09:15:21 +0000 (0:00:00.226)       0:09:24.607 ********
2026-03-14 09:15:22.119377 | controller | Saturday 14 March 2026  09:15:21 +0000 (0:00:00.226)       0:09:24.606 ********
2026-03-14 09:15:22.119390 | controller | ok: [controller]
2026-03-14 09:16:23.369256 | controller |
2026-03-14 09:16:23.369321 | controller | TASK [run_hook : Run hook without retry - Deploy watcher operator] *************
2026-03-14 09:16:23.369330 | controller | Saturday 14 March 2026  09:15:22 +0000 (0:00:00.259)       0:09:24.867 ********
2026-03-14 09:16:23.369335 | controller | Saturday 14 March 2026  09:15:22 +0000 (0:00:00.259)       0:09:24.865 ********
2026-03-14 09:16:23.369344 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_004_run_hook_without_retry_deploy.log
2026-03-14 09:16:23.392035 | controller | changed: [controller]
2026-03-14 09:16:23.392080 | controller |
2026-03-14 09:16:23.392087 | controller | TASK [run_hook : Run hook with retry - Deploy watcher operator] ****************
2026-03-14 09:16:23.392092 | controller | Saturday 14 March 2026  09:16:23 +0000 (0:01:01.250)       0:10:26.117 ********
2026-03-14 09:16:23.392097 | controller | Saturday 14 March 2026  09:16:23 +0000 (0:01:01.250)       0:10:26.116 ********
2026-03-14 09:16:23.392107 | controller | skipping: [controller]
2026-03-14 09:16:23.626531 | controller |
2026-03-14 09:16:23.626594 | controller | TASK [run_hook : Check if we have a file path={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] ***
2026-03-14 09:16:23.626601 | controller | Saturday 14 March 2026  09:16:23 +0000 (0:00:00.022)       0:10:26.140 ********
2026-03-14 09:16:23.626605 | controller | Saturday 14 March 2026  09:16:23 +0000 (0:00:00.022)       0:10:26.138 ********
2026-03-14 09:16:23.626619 | controller | ok: [controller]
2026-03-14 09:16:23.656394 | controller |
2026-03-14 09:16:23.656459 | controller | TASK [run_hook : Load generated content in main playbook file={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] ***
2026-03-14 09:16:23.656466 | controller | Saturday 14 March 2026  09:16:23 +0000 (0:00:00.234)       0:10:26.374 ********
2026-03-14 09:16:23.656471 | controller | Saturday 14 March 2026  09:16:23 +0000 (0:00:00.234)       0:10:26.373 ********
2026-03-14 09:16:23.656485 | controller | skipping: [controller]
2026-03-14 09:19:16.118131 | controller |
2026-03-14 09:19:16.118215 | controller | TASK [install kuttl test_suite dependencies chdir={{ ansible_user_dir }}/{{ operator_basedir }}, _raw_params=make kuttl-test-prep] ***
2026-03-14 09:19:16.118231 | controller | Saturday 14 March 2026  09:16:23 +0000 (0:00:00.030)       0:10:26.404 ********
2026-03-14 09:19:16.118243 | controller | Saturday 14 March 2026  09:16:23 +0000 (0:00:00.030)       0:10:26.403 ********
2026-03-14 09:19:16.118257 | controller | changed: [controller]
2026-03-14 09:19:16.201670 | controller |
2026-03-14 09:19:16.201771 | controller | PLAY RECAP *********************************************************************
2026-03-14 09:19:16.201790 | controller | controller                 : ok=117  changed=40   unreachable=0    failed=0    skipped=80   rescued=0    ignored=1
2026-03-14 09:19:16.201804 | controller |
2026-03-14 09:19:16.201818 | controller | Saturday 14 March 2026  09:19:16 +0000 (0:02:52.460)       0:13:18.864 ********
2026-03-14 09:19:16.201832 | controller | ===============================================================================
2026-03-14 09:19:16.201846 | controller | install_yamls_makes : Run openstack_init ------------------------------ 330.70s
2026-03-14 09:19:16.201860 | controller | install kuttl test_suite dependencies --------------------------------- 172.46s
2026-03-14 09:19:16.201874 | controller | run_hook : Run hook without retry - Deploy watcher operator ------------ 61.25s
2026-03-14 09:19:16.201887 | controller | ci_setup : Install needed packages ------------------------------------- 60.52s
2026-03-14 09:19:16.201900 | controller | openshift_obs : Wait for observability operator deployment ------------- 51.88s
2026-03-14 09:19:16.201913 | controller | install_yamls_makes : Run download_tools ------------------------------- 39.29s
2026-03-14 09:19:16.201927 | controller | run_hook : Run hook without retry - Download needed tools -------------- 33.81s
2026-03-14 09:19:16.201941 | controller | repo_setup : Initialize python venv and install requirements ------------ 8.43s
2026-03-14 09:19:16.201968 | controller | ci_setup : Install openshift client ------------------------------------- 5.26s
2026-03-14 09:19:16.201982 | controller | openshift_setup : Create required namespaces ---------------------------- 1.78s
2026-03-14 09:19:16.201995 | controller | openshift_obs : Install cluster observability operator. ----------------- 1.49s
2026-03-14 09:19:16.202008 | controller | install_yamls : Create the install_yamls parameters file ---------------- 1.22s
2026-03-14 09:19:16.202022 | controller | Gathering Facts --------------------------------------------------------- 1.17s
2026-03-14 09:19:16.202036 | controller | ci_setup : Manage directories ------------------------------------------- 1.11s
2026-03-14 09:19:16.202050 | controller | openshift_setup : Patch samples registry configuration ------------------ 0.98s
2026-03-14 09:19:16.202064 | controller | repo_setup : Make sure git-core package is installed -------------------- 0.96s
2026-03-14 09:19:16.202077 | controller | openshift_setup : Gather network.operator info -------------------------- 0.94s
2026-03-14 09:19:16.202090 | controller | repo_setup : Get repo-setup repository ---------------------------------- 0.93s
2026-03-14 09:19:16.202104 | controller | openshift_setup : Patch network operator -------------------------------- 0.89s
2026-03-14 09:19:16.202117 | controller | repo_setup : Install repo-setup package --------------------------------- 0.86s
2026-03-14 09:19:16.202130 | controller | Saturday 14 March 2026  09:19:16 +0000 (0:02:52.461)       0:13:18.864 ********
2026-03-14 09:19:16.202144 | controller | ===============================================================================
2026-03-14 09:19:16.202158 | controller | install_yamls_makes --------------------------------------------------- 370.09s
2026-03-14 09:19:16.202172 | controller | ansible.builtin.command ----------------------------------------------- 172.85s
2026-03-14 09:19:16.202185 | controller | run_hook --------------------------------------------------------------- 98.53s
2026-03-14 09:19:16.202198 | controller | ci_setup --------------------------------------------------------------- 68.20s
2026-03-14 09:19:16.202212 | controller | openshift_obs ---------------------------------------------------------- 54.09s
2026-03-14 09:19:16.202225 | controller | repo_setup ------------------------------------------------------------- 16.73s
2026-03-14 09:19:16.202239 | controller | openshift_setup --------------------------------------------------------- 5.16s
2026-03-14 09:19:16.202253 | controller | openshift_login --------------------------------------------------------- 4.30s
2026-03-14 09:19:16.202266 | controller | install_yamls ----------------------------------------------------------- 4.08s
2026-03-14 09:19:16.202280 | controller | cifmw_setup ------------------------------------------------------------- 1.62s
2026-03-14 09:19:16.202325 | controller | gather_facts ------------------------------------------------------------ 1.17s
2026-03-14 09:19:16.202345 | controller | discover_latest_image --------------------------------------------------- 0.88s
2026-03-14 09:19:16.202359 | controller | install_ca -------------------------------------------------------------- 0.44s
2026-03-14 09:19:16.202373 | controller | networking_mapper ------------------------------------------------------- 0.30s
2026-03-14 09:19:16.202499 | controller | ansible.builtin.set_fact ------------------------------------------------ 0.17s
2026-03-14 09:19:16.202521 | controller | ansible.builtin.include_role -------------------------------------------- 0.09s
2026-03-14 09:19:16.202534 | controller | ansible.builtin.include_vars -------------------------------------------- 0.06s
2026-03-14 09:19:16.202548 | controller | ansible.builtin.assert -------------------------------------------------- 0.05s
2026-03-14 09:19:16.202562 | controller | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2026-03-14 09:19:16.202576 | controller | total ----------------------------------------------------------------- 798.81s
2026-03-14 09:19:16.261440 | controller | changed
2026-03-14 09:19:16.341766 | 
2026-03-14 09:19:16.342046 | PLAY RECAP
2026-03-14 09:19:16.342100 | controller | ok: 1 changed: 1 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0
2026-03-14 09:19:16.342136 | 
2026-03-14 09:19:16.446159 | PRE-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/kuttl-from-operator-deps.yaml@main]
2026-03-14 09:19:16.450460 | RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/kuttl-from-operator-run.yaml@main]
2026-03-14 09:19:17.095584 | 
2026-03-14 09:19:17.095696 | PLAY [Run ci/playbooks/kuttl/run-kuttl-from-operator-targets.yaml]
2026-03-14 09:19:17.119579 | 
2026-03-14 09:19:17.119714 | TASK [Run kuttl tests from operator playbook]
2026-03-14 09:19:18.352544 | controller |
2026-03-14 09:19:18.386882 | controller | PLAY [controller] **************************************************************
2026-03-14 09:19:18.386961 | controller |
2026-03-14 09:19:18.386971 | controller | TASK [Run hooks before running kuttl tests name=run_hook] **********************
2026-03-14 09:19:18.386978 | controller | Saturday 14 March 2026  09:19:18 +0000 (0:00:00.046)       0:00:00.046 ********
2026-03-14 09:19:18.386986 | controller | Saturday 14 March 2026  09:19:18 +0000 (0:00:00.045)       0:00:00.045 ********
2026-03-14 09:19:18.387004 | controller |
2026-03-14 09:19:18.455959 | controller | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] ***
2026-03-14 09:19:18.456034 | controller | Saturday 14 March 2026  09:19:18 +0000 (0:00:00.034)       0:00:00.081 ********
2026-03-14 09:19:18.456042 | controller | Saturday 14 March 2026  09:19:18 +0000 (0:00:00.034)       0:00:00.079 ********
2026-03-14 09:19:18.456056 | controller | ok: [controller]
2026-03-14 09:19:18.562820 | controller |
2026-03-14 09:19:18.562878 | controller | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] ***
2026-03-14 09:19:18.562892 | controller | Saturday 14 March 2026  09:19:18 +0000 (0:00:00.069)       0:00:00.150 ********
2026-03-14 09:19:18.562898 | controller | Saturday 14 March 2026  09:19:18 +0000 (0:00:00.069)       0:00:00.149 ********
2026-03-14 09:19:18.562910 | controller | ok: [controller]
2026-03-14 09:19:18.708485 | controller |
2026-03-14 09:19:18.708550 | controller | TASK [run_hook : Loop on hooks for pre_kuttl_from_operator _raw_params={{ hook.type }}.yml] ***
2026-03-14 09:19:18.708558 | controller | Saturday 14 March 2026  09:19:18 +0000 (0:00:00.106)       0:00:00.257 ********
2026-03-14 09:19:18.708563 | controller | Saturday 14 March 2026  09:19:18 +0000 (0:00:00.106)       0:00:00.255 ********
2026-03-14 09:19:18.708578 | controller | skipping: [controller]
2026-03-14 09:44:27.066501 | controller |
2026-03-14 09:44:27.066579 | controller | TASK [run kuttl test suite from operator Makefile chdir={{ ansible_user_dir }}/{{ operator_basedir }}, _raw_params=make kuttl-test-run] ***
2026-03-14 09:44:27.066588 | controller | Saturday 14 March 2026  09:19:18 +0000 (0:00:00.145)       0:00:00.403 ********
2026-03-14 09:44:27.066595 | controller | Saturday 14 March 2026  09:19:18 +0000 (0:00:00.145)       0:00:00.401 ********
2026-03-14 09:44:27.066610 | controller | task path: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/run-kuttl-from-operator-targets.yaml:16
2026-03-14 09:44:27.066653 | controller | fatal: [controller]: FAILED! =>
2026-03-14 09:44:27.066664 | controller |     changed: true
2026-03-14 09:44:27.066671 | controller |     cmd:
2026-03-14 09:44:27.066677 | controller |     - make
2026-03-14 09:44:27.066684 | controller |     - kuttl-test-run
2026-03-14 09:44:27.066691 | controller |     delta: '0:25:07.262437'
2026-03-14 09:44:27.066703 | controller |     end: '2026-03-14 09:44:26.323887'
2026-03-14 09:44:27.066712 | controller |     msg: non-zero return code
2026-03-14 09:44:27.066718 | controller |     rc: 2
2026-03-14 09:44:27.066724 | controller |     start: '2026-03-14 09:19:19.061450'
2026-03-14 09:44:27.066730 | controller |     stderr: |-
2026-03-14 09:44:27.066736 | controller |       Warning: spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`.
2026-03-14 09:44:27.066743 | controller |       Warning: The 'cinderBackup' field is deprecated and will be removed in a future release. Please migrate to 'cinderBackups'.
2026-03-14 09:44:27.066748 | controller |       make: *** [Makefile:444: kuttl-test-run] Error 1
2026-03-14 09:44:27.066755 | controller |     stderr_lines:
2026-03-14 09:44:27.066760 | controller |     - 'Warning: spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default
2026-03-14 09:44:27.066765 | controller |       value changed from `Never` to `Always`.'
2026-03-14 09:44:27.066770 | controller |     - 'Warning: The ''cinderBackup'' field is deprecated and will be removed in a future
2026-03-14 09:44:27.066774 | controller |       release. Please migrate to ''cinderBackups''.'
2026-03-14 09:44:27.066779 | controller |     - 'make: *** [Makefile:444: kuttl-test-run] Error 1'
2026-03-14 09:44:27.066795 | controller |     stdout: "oc kuttl test --v 1 --start-kind=false --config test/kuttl/test-suites/default/config.yaml\n===
2026-03-14 09:44:27.066801 | controller |       RUN   kuttl\n    harness.go:463: starting setup\n    harness.go:255: running tests
2026-03-14 09:44:27.066805 | controller |       using configured kubeconfig.\n    harness.go:278: Successful connection to cluster
2026-03-14 09:44:27.066809 | controller |       at: https://api.crc.testing:6443\n    harness.go:363: running tests\n    harness.go:75:
2026-03-14 09:44:27.066815 | controller |       going to run test suite with timeout of 300 seconds for each step\n    harness.go:375:
2026-03-14 09:44:27.066820 | controller |       testsuite: test/kuttl/test-suites/default/ has 11 tests\n=== RUN   kuttl/harness\n===
2026-03-14 09:44:27.066824 | controller |       RUN   kuttl/harness/appcred-tests\n=== PAUSE kuttl/harness/appcred-tests\n=== RUN
2026-03-14 09:44:27.066829 | controller |       \  kuttl/harness/common\n=== PAUSE kuttl/harness/common\n=== RUN   kuttl/harness/deps\n===
2026-03-14 09:44:27.066833 | controller |       PAUSE kuttl/harness/deps\n=== RUN   kuttl/harness/watcher\n=== PAUSE kuttl/harness/watcher\n===
2026-03-14 09:44:27.066837 | controller |       RUN   kuttl/harness/watcher-api-scaling\n=== PAUSE kuttl/harness/watcher-api-scaling\n===
2026-03-14 09:44:27.066841 | controller |       RUN   kuttl/harness/watcher-cinder\n=== PAUSE kuttl/harness/watcher-cinder\n===
2026-03-14 09:44:27.066846 | controller |       RUN   kuttl/harness/watcher-notification\n=== PAUSE kuttl/harness/watcher-notification\n===
2026-03-14 09:44:27.066850 | controller |       RUN   kuttl/harness/watcher-rmquser\n=== PAUSE kuttl/harness/watcher-rmquser\n===
2026-03-14 09:44:27.066854 | controller |       RUN   kuttl/harness/watcher-tls\n=== PAUSE kuttl/harness/watcher-tls\n=== RUN   kuttl/harness/watcher-tls-certs-change\n===
2026-03-14 09:44:27.066859 | controller |       PAUSE kuttl/harness/watcher-tls-certs-change\n=== RUN   kuttl/harness/watcher-topology\n===
2026-03-14 09:44:27.066863 | controller |       PAUSE kuttl/harness/watcher-topology\n=== CONT  kuttl/harness/appcred-tests\n    logger.go:42:
2026-03-14 09:44:27.066867 | controller |       09:19:19 | appcred-tests | Skipping creation of user-supplied namespace: watcher-kuttl-default\n
2026-03-14 09:44:27.066871 | controller |       \   logger.go:42: 09:19:19 | appcred-tests/0-cleanup | starting test step 0-cleanup\n
2026-03-14 09:44:27.066882 | controller |       \   logger.go:42: 09:19:19 | appcred-tests/0-cleanup | test step completed 0-cleanup\n
2026-03-14 09:44:27.066886 | controller |       \   logger.go:42: 09:19:19 | appcred-tests/1-deploy | starting test step 1-deploy\n
2026-03-14 09:44:27.066890 | controller |       \   logger.go:42: 09:19:19 | appcred-tests/1-deploy | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:44:27.066895 | controller |       created\n    logger.go:42: 09:19:19 | appcred-tests/1-deploy | running command:
2026-03-14 09:44:27.066899 | controller |       [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        oc wait -n \"${NS}\"
2026-03-14 09:44:27.066903 | controller |       watcher watcher-kuttl --for=condition=Ready --timeout=600s]\n    logger.go:42: 09:20:09
2026-03-14 09:44:27.066907 | controller |       | appcred-tests/1-deploy | watcher.watcher.openstack.org/watcher-kuttl condition
2026-03-14 09:44:27.066912 | controller |       met\n    logger.go:42: 09:20:09 | appcred-tests/1-deploy | test step completed 1-deploy\n
2026-03-14 09:44:27.066916 | controller |       \   logger.go:42: 09:20:09 | appcred-tests/2-deploy-appcred | starting test step
2026-03-14 09:44:27.066920 | controller |       2-deploy-appcred\n    logger.go:42: 09:20:09 | appcred-tests/2-deploy-appcred |
2026-03-14 09:44:27.066924 | controller |       running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.066929 | controller |       \       old_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.066933 | controller |       \       old_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.066938 | controller |       \       old_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.066942 | controller |       jsonpath='{.metadata.uid}')\n        oc create configmap appcred-watcher-pre \\\n
2026-03-14 09:44:27.066954 | controller |       \         --from-literal=api_uid=\"${old_api_uid}\" \\\n          --from-literal=applier_uid=\"${old_applier_uid}\"
2026-03-14 09:44:27.066958 | controller |       \\\n          --from-literal=decision_engine_uid=\"${old_de_uid}\" \\\n          --dry-run=client
2026-03-14 09:44:27.066963 | controller |       -o yaml | oc apply -n \"${NS}\" -f -\n        \n        cat <<'EOF' | oc apply -n
2026-03-14 09:44:27.066967 | controller |       \"${NS}\" -f -\n        apiVersion: keystone.openstack.org/v1beta1\n        kind:
2026-03-14 09:44:27.066971 | controller |       KeystoneApplicationCredential\n        metadata:\n          name: ac-watcher\n        spec:\n
2026-03-14 09:44:27.066975 | controller |       \         secret: osp-secret\n          passwordSelector: WatcherPassword\n          userName:
2026-03-14 09:44:27.066980 | controller |       watcher\n          roles:\n            - admin\n            - service\n          unrestricted:
2026-03-14 09:44:27.066984 | controller |       false\n        EOF\n        \n        oc patch watcher watcher-kuttl -n \"${NS}\"
2026-03-14 09:44:27.066988 | controller |       --type=merge -p '{\"spec\":{\"auth\":{\"applicationCredentialSecret\":\"ac-watcher-secret\"}}}']\n
2026-03-14 09:44:27.066993 | controller |       \   logger.go:42: 09:20:10 | appcred-tests/2-deploy-appcred | configmap/appcred-watcher-pre
2026-03-14 09:44:27.066997 | controller |       created\n    logger.go:42: 09:20:10 | appcred-tests/2-deploy-appcred | keystoneapplicationcredential.keystone.openstack.org/ac-watcher
2026-03-14 09:44:27.067002 | controller |       created\n    logger.go:42: 09:20:10 | appcred-tests/2-deploy-appcred | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:44:27.067006 | controller |       patched\n    logger.go:42: 09:20:10 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.067010 | controller |       [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n        echo \"Waiting
2026-03-14 09:44:27.067014 | controller |       for KeystoneApplicationCredential ac-watcher to be Ready...\"\n        oc wait -n
2026-03-14 09:44:27.067019 | controller |       \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n        \n        ac_id=$(oc
2026-03-14 09:44:27.067023 | controller |       get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n        if [
2026-03-14 09:44:27.067029 | controller |       -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.067034 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.067038 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.067043 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.067047 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.067051 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.067056 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.067060 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.067064 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.067068 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.067074 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.067079 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.067083 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.067234 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.067238 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.067245 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.067249 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.067253 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.067258 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.067262 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.067266 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.067271 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.067275 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.067279 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.067285 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.067316 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.067320 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.067325 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.067330 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.067334 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.067338 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.067343 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.067347 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:20:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.067352 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.067356 | controller |       09:25:10 | appcred-tests/2-deploy-appcred | error: timed out waiting for the condition
2026-03-14 09:44:27.067360 | controller |       on keystoneapplicationcredentials/ac-watcher\n    logger.go:42: 09:25:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.067364 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.067369 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.067373 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.067377 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.067381 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.067386 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.067390 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.067394 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.067398 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.067403 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.067407 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.067411 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.067420 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.067425 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.067429 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.067433 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.067438 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.067442 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.067447 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.067453 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.067459 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.067466 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.067472 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.067478 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.067483 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.067489 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.067494 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.067498 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.067502 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.067507 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.067521 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.067526 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.067530 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.067535 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.067539 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.067544 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.067677 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.067682 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.067686 | controller |       09:25:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.067690 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.067695 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.067699 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.067703 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.067711 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.067715 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.067720 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.067724 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.067728 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.067732 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.067737 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.067741 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.067745 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.067749 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.067754 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.067760 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.067766 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.067772 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.067779 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.067785 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.067792 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.067798 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.067805 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.067812 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.067819 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.067826 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.067832 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.067839 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.067846 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.067852 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.067861 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.067865 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.067880 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.067885 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.067890 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.067896 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.067900 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.067907 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.067912 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.067916 | controller |       09:25:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.067920 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.067925 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.067929 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.067933 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.067937 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.067942 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.067946 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.067950 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.067954 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.067959 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.067963 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.067967 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.067971 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.067976 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.067980 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.067984 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.067988 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.067993 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.067997 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.068002 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.068007 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.068011 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.068015 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.068020 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.068024 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.068028 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.068032 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.068037 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.068041 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.068047 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.068052 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.068056 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.068060 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.068072 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.068076 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.068080 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.068085 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.068089 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.068093 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.068098 | controller |       09:25:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.068102 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.068106 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.068110 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.068114 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.068119 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.068123 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.068127 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.068131 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.068136 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.068140 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.068145 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.068149 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.068155 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.068160 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.068164 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.068168 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.068173 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.068177 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.068181 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.068186 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.068192 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.068196 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.068201 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.068205 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.068209 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.068213 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.068218 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.068222 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.068226 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.068230 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.068235 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.068239 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.068243 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.068247 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.068252 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.068268 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.068272 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.068277 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.068282 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.068289 | controller |       09:25:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.068315 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.068320 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.068324 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.068328 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.068346 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.068351 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.068355 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.068359 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.068364 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.068368 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.068372 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.068377 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.068381 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.068388 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.068392 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.068396 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.068408 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.068412 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.068417 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.068421 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.068426 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.068430 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.068434 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.068439 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.068443 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.068448 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.068454 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.068460 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.068467 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.068473 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.068479 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.068485 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.068493 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.068497 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.068502 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.068506 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.068519 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.068523 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.068528 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.068532 | controller |       09:25:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.068537 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.068541 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.068545 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.068550 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.068557 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.068561 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.068565 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.068570 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.068574 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.068578 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.068582 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.068587 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.068591 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.068595 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.068599 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.068604 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.068608 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.068612 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.068618 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.068623 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.068627 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.068631 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.068636 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.068640 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.068644 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.068648 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.068653 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.068657 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.068661 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.068666 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.068670 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.068674 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.068678 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.068683 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.068687 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.068691 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.068696 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.068702 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.068713 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.068718 | controller |       09:25:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.068722 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.068727 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.068731 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.068736 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.068741 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.068745 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.068749 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.068754 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.068758 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.068762 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.068766 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.068772 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.068777 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.068781 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.068785 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.068791 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.068795 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.068800 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.068804 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.068808 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.068812 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.068817 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.068821 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.068825 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.068829 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.068834 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.068838 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.068842 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.068846 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.068853 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.068857 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.068862 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.068866 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.068870 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.068874 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.068879 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.068883 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.068887 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.068892 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.068907 | controller |       09:25:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.068914 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.068921 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.068928 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.068936 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.068943 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.068950 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.068957 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.068965 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.068972 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.068980 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.068986 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.068992 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.068997 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.069002 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.069006 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.069010 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.069014 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.069019 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.069023 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.069028 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.069035 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.069040 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.069044 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.069048 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.069052 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.069057 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.069061 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.069065 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.069072 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.069076 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.069081 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.069085 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.069089 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.069094 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.069098 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.069102 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.069106 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.069111 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.069115 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.069119 | controller |       09:25:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.069133 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.069138 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.069142 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.069147 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.069151 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.069156 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.069160 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.069164 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.069168 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.069173 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.069177 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.069181 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.069185 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.069192 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.069196 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.069201 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.069205 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.069209 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.069213 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.069218 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.069222 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.069226 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.069230 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.069235 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.069239 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.069243 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.069248 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.069252 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.069256 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.069260 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.069266 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.069271 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.069275 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.069279 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.069284 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.069345 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.069354 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.069359 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.069364 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.069369 | controller |       09:25:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.069373 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.069377 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.069412 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.069419 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.069429 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.069436 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.069442 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.069446 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.069451 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.069455 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.069460 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.069464 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.069468 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.069473 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.069477 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.069481 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.069487 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.069492 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.069496 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.069501 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.069505 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.069513 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.069517 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.069522 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.069526 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.069530 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.069534 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.069539 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.069543 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.069549 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.069553 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.069558 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.069562 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.069566 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.069571 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.069575 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.069579 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.069589 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.069594 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.069599 | controller |       09:25:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.069603 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.069607 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.069611 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.069623 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.069628 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.069632 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.069637 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.069641 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.069646 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.069650 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.069654 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.069658 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.069663 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.069667 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.069671 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.069675 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.069680 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.069684 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.069688 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.069693 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.069697 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.069701 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.069705 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.069710 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.069714 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.069718 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.069723 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.069729 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.069733 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.069739 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.069743 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.069747 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.069752 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.069756 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.069760 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.069764 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.069769 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.069773 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.069777 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.069782 | controller |       09:25:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.069786 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.069790 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.069794 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.069799 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.069811 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.069815 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.069820 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.069824 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.069828 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.069833 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.069837 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.069841 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.069845 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.069849 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.069854 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.069858 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.069862 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.069866 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.069871 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.069875 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.069881 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.069885 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.069890 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.069894 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.069898 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.069902 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.069907 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.069911 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.069915 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.069919 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.069923 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.069928 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.069932 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.069936 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.069941 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.069945 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.069949 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.069953 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.069958 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.069962 | controller |       09:25:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.069966 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.069970 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.069975 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.069979 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.069983 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.069987 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.069998 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.070003 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.070007 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.070013 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.070017 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.070022 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.070026 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.070034 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.070040 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.070047 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.070053 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.070059 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.070065 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.070072 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.070079 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.070086 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.070093 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.070100 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.070106 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.070113 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.070120 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.070126 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.070132 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.070137 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.070141 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.070146 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.070150 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.070154 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.070158 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.070163 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.070167 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.070171 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.070176 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.070180 | controller |       09:25:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.070184 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.070188 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.070193 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.070197 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.070204 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.070208 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.070212 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.070226 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.070231 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.070235 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.070239 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.070244 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.070248 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.070252 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.070256 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.070263 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.070267 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.070271 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.070275 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.070280 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.070284 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.070303 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.070311 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.070315 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.070319 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.070325 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.070329 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.070334 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.070338 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.070342 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.070346 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.070351 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.070355 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.070359 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.070364 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.070368 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.070372 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.070378 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.070383 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.070387 | controller |       09:25:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.070391 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.070395 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.070400 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.070404 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.070408 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.070413 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.070417 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.070421 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.070433 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.070437 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.070442 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.070446 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.070451 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.070455 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.070459 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.070464 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.070468 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.070472 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.070476 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.070481 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.070486 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.070490 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.070495 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.070499 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.070503 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.070518 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.070522 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.070526 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.070538 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.070544 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.070549 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.070553 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.070557 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.070562 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.070566 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.070570 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.070574 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.070579 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.070590 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.070594 | controller |       09:25:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.070599 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.070640 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.070648 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.070652 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.070656 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.070661 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.070665 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.070669 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.070675 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.070680 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.070713 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.070720 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.070724 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.070728 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.070733 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.070737 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.070762 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.070766 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.070770 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.070775 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.070782 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.070786 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.070791 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.070795 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.070799 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.070804 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.070808 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.070812 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.070816 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.070821 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.070827 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.070831 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.070836 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.070840 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.070844 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.070849 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.070853 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.070857 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.070861 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.070866 | controller |       09:25:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.070870 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.070874 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.070878 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.070883 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.070887 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.070891 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.070896 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.070900 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.070904 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.070909 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.070913 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.070926 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.070933 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.070941 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.070947 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.070953 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.070957 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.070961 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.070966 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.070970 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.070974 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.070979 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.070983 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.070987 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.070993 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.070997 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.071002 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.071006 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.071010 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.071014 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.071019 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.071023 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.071027 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.071040 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.071045 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.071049 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.071053 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.071058 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.071062 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.071066 | controller |       09:25:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.071071 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.071075 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.071079 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.071083 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.071089 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.071093 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.071098 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.071102 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.071106 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.071111 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.071115 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.071119 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.071123 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.071135 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.071140 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.071144 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.071148 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.071153 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.071157 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.071161 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.071166 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.071172 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.071178 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.071184 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.071190 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.071196 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.071202 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.071209 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.071216 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.071223 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.071229 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.071236 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.071243 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.071250 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.071257 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.071264 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.071270 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.071281 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.071302 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.071313 | controller |       09:25:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.071319 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.071324 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.071328 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.071334 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.071339 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.071344 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.071348 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.071352 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.071357 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.071361 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.071365 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.071370 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.071374 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.071378 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.071392 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.071397 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.071401 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.071406 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.071410 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.071414 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.071419 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.071423 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.071427 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.071432 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.071436 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.071440 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.071444 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.071449 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.071453 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.071460 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.071465 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.071469 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.071473 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.071478 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.071482 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.071486 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.071490 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.071495 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.071499 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.071503 | controller |       09:25:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.071507 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.071512 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.071516 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.071520 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.071524 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.071529 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.071533 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.071537 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.071541 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.071546 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.071550 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.071554 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.071558 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.071562 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.071567 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.071571 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.071582 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.071586 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.071591 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.071595 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.071602 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.071606 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.071611 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.071616 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.071621 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.071625 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.071629 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.071665 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.071670 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.071674 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.071678 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.071683 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.071687 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.071691 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.071695 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.071700 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.071704 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.071708 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.071713 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.071717 | controller |       09:25:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.071721 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.071725 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.071730 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.071734 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.071738 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.071743 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.071747 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.071751 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.071755 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.071760 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.071764 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.071768 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.071773 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.071779 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.071783 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.071789 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.071793 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.071805 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.071810 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.071814 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.071819 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.071823 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.071827 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.071832 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.071836 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.071840 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.071845 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.071849 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.071853 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.071858 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.071862 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.071866 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.071871 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.071875 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.071879 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.071883 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.071888 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.071892 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.071898 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.071902 | controller |       09:25:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.071906 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.071913 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.071918 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.071922 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.071928 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.071934 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.071938 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.071942 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.071947 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.071951 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.071955 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.071959 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.071964 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.071968 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.071972 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.071976 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.071981 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.071985 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.071996 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.072001 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.072005 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.072010 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.072014 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.072018 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.072022 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.072027 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.072031 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.072035 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.072039 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.072044 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.072048 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.072052 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.072056 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.072061 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.072065 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.072069 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.072073 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.072079 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.072084 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.072088 | controller |       09:25:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.072092 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.072096 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.072101 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.072105 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.072109 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.072113 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.072118 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.072122 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.072126 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.072130 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.072135 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.072139 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.072144 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.072149 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.072153 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.072157 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.072161 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.072165 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.072170 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.072182 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.072186 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.072191 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.072195 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.072199 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.072204 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.072208 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.072212 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.072216 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.072220 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.072227 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.072233 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.072237 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.072241 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.072245 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.072250 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.072254 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.072258 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.072262 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.072267 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.072271 | controller |       09:25:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.072275 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.072279 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.072283 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.072290 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.072309 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.072314 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.072320 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.072326 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.072332 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.072338 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.072344 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.072351 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.072357 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.072364 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.072371 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.072377 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.072384 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.072391 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.072397 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.072404 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.072414 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.072430 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.072435 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.072439 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.072444 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.072448 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.072452 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.072457 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.072462 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.072466 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.072470 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.072474 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.072478 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.072483 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.072487 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.072491 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.072495 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.072500 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.072504 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.072508 | controller |       09:25:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.072512 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.072519 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.072523 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.072527 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.072532 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.072536 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.072540 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.072544 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.072548 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.072553 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.072557 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.072561 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.072566 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.072572 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.072577 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.072581 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.072585 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.072589 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.072593 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.072598 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.072603 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.072607 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.072618 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.072623 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.072627 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.072631 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.072635 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.072639 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.072644 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.072648 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.072652 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.072656 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.072660 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.072664 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.072669 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.072673 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.072677 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.072681 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.072685 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.072690 | controller |       09:25:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.072694 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.072698 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.072702 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.072706 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.072714 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.072718 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.072722 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.072727 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.072731 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.072735 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.072739 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.072743 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.072747 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.072751 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.072756 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.072760 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.072765 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.072769 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.072773 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.072778 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.072782 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.072786 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.072790 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.072795 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.072805 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.072809 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.072814 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.072818 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.072822 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.072826 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.072831 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.072835 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.072839 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.072843 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.072848 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.072852 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.072856 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.072862 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.072866 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.072871 | controller |       09:25:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.072875 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.072879 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.072883 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.072888 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.072892 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.072896 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.072900 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.072905 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.072909 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.072913 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.072917 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.072921 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.072926 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.072930 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.072934 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.072938 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.072942 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.072947 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.072951 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.072955 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.072959 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.072964 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.072968 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.072974 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.072979 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.072990 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.072995 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.073000 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.073004 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.073011 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.073016 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.073020 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.073024 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.073029 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.073033 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.073037 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.073042 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.073046 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.073050 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.073054 | controller |       09:25:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.073061 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.073065 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.073069 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.073074 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.073078 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.073082 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.073087 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.073091 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.073095 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.073099 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.073104 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.073108 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.073112 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.073116 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.073121 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.073126 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.073130 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.073135 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.073139 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.073143 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.073149 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.073153 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.073158 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.073162 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.073166 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.073171 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.073175 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.073185 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.073189 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.073194 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.073198 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.073202 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.073206 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.073211 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.073215 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.073219 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.073224 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.073228 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.073232 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.073237 | controller |       09:25:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.073241 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.073245 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.073249 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.073253 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.073258 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.073262 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.073266 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.073270 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.073274 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.073279 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.073283 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.073290 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.073311 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.073318 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.073323 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.073328 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.073332 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.073336 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.073341 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.073345 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.073349 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.073353 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.073357 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.073362 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.073366 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.073370 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.073374 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.073378 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.073389 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.073393 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.073421 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.073427 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.073433 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.073438 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.073444 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.073450 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.073456 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.073461 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.073467 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.073473 | controller |       09:25:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.073478 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.073485 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.073491 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.073497 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.073508 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.073516 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.073522 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.073529 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.073535 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.073542 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.073549 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.073556 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.073562 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.073568 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.073574 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.073580 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.073586 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.073591 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.073597 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.073602 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.073610 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.073614 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.073619 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.073623 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.073627 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.073631 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.073636 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.073640 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.073644 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.073658 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.073663 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.073667 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.073672 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.073676 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.073680 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.073684 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.073688 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.073695 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.073700 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.073704 | controller |       09:25:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.073709 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.073713 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.073717 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.073757 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.073761 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.073765 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.073770 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.073774 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.073778 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.073782 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.073786 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.073791 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.073795 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.073799 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.073803 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.073808 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.073812 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.073816 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.073820 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.073825 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.073829 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.073833 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.073837 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.073841 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.073846 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.073850 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.073854 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.073858 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.073862 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.073868 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.073873 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.073884 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.073888 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.073893 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.073897 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.073901 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.073905 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.073910 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.073914 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.073918 | controller |       09:25:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.073922 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.073926 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.073931 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.073935 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.073939 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.073943 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.073947 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.073951 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.073956 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.073960 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.073964 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.073968 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.073972 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.073978 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.073982 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.073986 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.073991 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.073995 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.073999 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.074003 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.074009 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.074014 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.074018 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.074022 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.074026 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.074030 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.074035 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.074039 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.074043 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.074047 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.074051 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.074055 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.074066 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.074071 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.074075 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.074079 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.074083 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.074088 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.074092 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.074096 | controller |       09:25:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.074100 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.074105 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.074109 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.074113 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.074119 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.074124 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.074128 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.074132 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.074136 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.074141 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.074147 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.074153 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.074159 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.074166 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.074170 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.074174 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.074179 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.074183 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.074187 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.074191 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.074196 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.074200 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.074204 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.074208 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.074212 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.074216 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.074221 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.074225 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.074229 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.074233 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.074237 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.074242 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.074246 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.074258 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.074262 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.074268 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.074272 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.074277 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.074281 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.074285 | controller |       09:25:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.074367 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.074377 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.074382 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.074386 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.074392 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.074396 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.074401 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.074427 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.074432 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.074456 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.074460 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.074465 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.074469 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.074473 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.074477 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.074481 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.074486 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.074490 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.074520 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.074528 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.074534 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.074540 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.074545 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.074551 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.074556 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.074562 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.074567 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.074572 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.074576 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.074580 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.074584 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.074588 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.074592 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.074597 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.074601 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.074617 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.074624 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.074634 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.074640 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.074647 | controller |       09:25:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.074654 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.074660 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.074667 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.074673 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.074680 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.074686 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.074693 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.074700 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.074706 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.074712 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.074718 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.074724 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.074732 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.074737 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.074743 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.074749 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.074755 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.074760 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.074766 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.074770 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.074775 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.074780 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.074784 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.074788 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.074793 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.074797 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.074801 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.074805 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.074810 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.074816 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.074820 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.074824 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.074829 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.074833 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.074837 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.074841 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.074855 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.074860 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.074864 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.074869 | controller |       09:25:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.074873 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.074877 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.074881 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.074886 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.074890 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.074894 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.074898 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.074903 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.074908 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.074912 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.074916 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.074920 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.074924 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.074928 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.074933 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.074937 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.074941 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.074945 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.074950 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.074954 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.074959 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.074964 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.074968 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.074972 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.074976 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.074980 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.074985 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.074989 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.074993 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.074997 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.075001 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.075006 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.075011 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.075015 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.075019 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.075024 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.075028 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.075039 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.075043 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.075047 | controller |       09:25:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.075052 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.075056 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.075060 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.075065 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.075069 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.075073 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.075077 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.075081 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.075085 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.075090 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.075094 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.075098 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.075102 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.075111 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.075115 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.075119 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.075123 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.075128 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.075132 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.075136 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.075140 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.075145 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.075149 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.075153 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.075157 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.075161 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.075165 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.075170 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.075174 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.075178 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.075183 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.075188 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.075192 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.075196 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.075200 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.075204 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.075209 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.075213 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.075217 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.075228 | controller |       09:25:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.075233 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.075237 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.075241 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.075246 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.075284 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.075304 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.075313 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.075317 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.075321 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.075325 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.075330 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.075335 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.075339 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.075344 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.075348 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.075352 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.075356 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.075360 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.075365 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.075369 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.075373 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.075377 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.075381 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.075385 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.075390 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.075394 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.075398 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.075402 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.075406 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.075410 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.075415 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.075419 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.075423 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.075427 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.075431 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.075435 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.075440 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.075446 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.075450 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.075454 | controller |       09:25:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.075466 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.075470 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.075475 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.075479 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.075483 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.075488 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.075492 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.075496 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.075500 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.075504 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.075509 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.075513 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.075517 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.075521 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.075525 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.075529 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.075534 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.075538 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.075542 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.075546 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.075551 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.075555 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.075559 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.075563 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.075568 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.075572 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.075576 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.075580 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.075587 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.075593 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.075597 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.075602 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.075606 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.075610 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.075614 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.075619 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.075624 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.075628 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.075632 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.075637 | controller |       09:25:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.075641 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.075652 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.075657 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.075661 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.075679 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.075684 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.075688 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.075692 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.075697 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.075701 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.075705 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.075709 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.075714 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.075718 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.075722 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.075726 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.075731 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.075735 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.075739 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.075744 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.075756 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.075762 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.075769 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.075775 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.075781 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.075788 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.075795 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.075803 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.075810 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.075817 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.075824 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.075831 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.075838 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.075845 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.075852 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.075858 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.075862 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.075866 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.075871 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.075875 | controller |       09:25:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.075880 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.075885 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.075889 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.075903 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.075907 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.075912 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.075916 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.075920 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.075925 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.075929 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.075936 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.075940 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.075944 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.075951 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.075955 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.075959 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.075963 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.075967 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.075972 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.075976 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.075980 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.075984 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.075989 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.075993 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.075997 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.076001 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.076005 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.076009 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.076014 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.076018 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.076022 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.076026 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.076030 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.076035 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.076039 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.076043 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.076047 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.076051 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.076055 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.076060 | controller |       09:25:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.076064 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.076068 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.076073 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.076077 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.076090 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.076095 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.076099 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.076103 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.076108 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.076112 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.076116 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.076120 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.076125 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.076129 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.076133 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.076139 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.076143 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.076148 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.076152 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.076156 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.076160 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.076164 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.076169 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.076173 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.076177 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.076181 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.076185 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.076191 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.076195 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.076200 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.076204 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.076208 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.076212 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.076216 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.076221 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.076225 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.076229 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.076236 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.076240 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.076244 | controller |       09:25:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.076249 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:25:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.076253 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.076257 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.076261 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.076265 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.076270 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.076280 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.076285 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.076308 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.076317 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.076321 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.076325 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.076329 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.076334 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.076338 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.076342 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.076346 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.076351 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.076355 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.076359 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.076363 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.076367 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.076372 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.076376 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.076380 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.076384 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.076388 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.076393 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.076397 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.076403 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.076407 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.076412 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.076416 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.076420 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.076424 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.076429 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.076433 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.076437 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:25:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.076441 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.076445 | controller |       09:25:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.076449 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.076454 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.076458 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.076462 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.076466 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.076471 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.076475 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.076486 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.076491 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.076497 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.076501 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.076506 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.076510 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.076514 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.076518 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.076522 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.076527 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.076531 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.076535 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.076539 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.076545 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.076549 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.076553 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.076558 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.076562 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.076566 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.076570 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.076574 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.076579 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.076583 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.076589 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.076593 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.076597 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.076601 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.076606 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.076610 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.076614 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.076618 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.076624 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.076628 | controller |       09:26:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.076632 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.076637 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.076641 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.076645 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.076649 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.076654 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.076658 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.076662 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.076673 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.076677 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.076681 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.076686 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.076690 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.076696 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.076700 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.076704 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.076708 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.076712 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.076717 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.076721 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.076726 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.076731 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.076735 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.076739 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.076743 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.076751 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.076756 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.076760 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.076764 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.076768 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.076772 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.076777 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.076781 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.076785 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.076789 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.076794 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.076798 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.076802 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.076806 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.076810 | controller |       09:26:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.076815 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.076819 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.076823 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.076827 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.076833 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.076837 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.076841 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.076846 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.076850 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.076854 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.076865 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.076869 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.076873 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.076878 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.076882 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.076886 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.076892 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.076898 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.076903 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.076909 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.076916 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.076922 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.076928 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.076936 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.076943 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.076950 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.076957 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.076965 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.076972 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.076979 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.076986 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.076992 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.076997 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.077002 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.077007 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.077011 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.077015 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.077022 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.077027 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.077031 | controller |       09:26:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.077036 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.077040 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.077044 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.077048 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.077055 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.077059 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.077064 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.077068 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.077074 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.077078 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.077082 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.077097 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.077102 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.077106 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.077111 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.077115 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.077120 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.077124 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.077128 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.077132 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.077137 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.077141 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.077145 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.077149 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.077154 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.077159 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.077163 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.077167 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.077172 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.077178 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.077182 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.077186 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.077191 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.077195 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.077199 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.077203 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.077208 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.077212 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.077216 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.077221 | controller |       09:26:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.077225 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.077229 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.077233 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.077238 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.077242 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.077248 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.077253 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.077257 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.077261 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.077265 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.077270 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.077274 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.077278 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.077304 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.077313 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.077318 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.077322 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.077326 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.077331 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.077336 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.077345 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.077351 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.077357 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.077363 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.077369 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.077375 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.077379 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.077383 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.077387 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.077392 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.077396 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.077400 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.077405 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.077450 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.077455 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.077460 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.077464 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.077468 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.077472 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.077476 | controller |       09:26:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.077481 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.077485 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.077489 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.077493 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.077497 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.077502 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.077506 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.077510 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.077514 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.077519 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.077523 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.077527 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.077531 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.077542 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.077554 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.077561 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.077565 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.077569 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.077573 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.077578 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.077582 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.077586 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.077590 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.077594 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.077599 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.077603 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.077607 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.077611 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.077615 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.077619 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.077624 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.077628 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.077632 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.077636 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.077640 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.077645 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.077649 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.077653 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.077657 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.077661 | controller |       09:26:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.077666 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.077670 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.077674 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.077680 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.077687 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.077691 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.077695 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.077700 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.077704 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.077708 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.077712 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.077716 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.077720 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.077725 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.077729 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.077733 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.077744 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.077748 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.077752 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.077757 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.077763 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.077767 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.077771 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.077776 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.077780 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.077784 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.077788 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.077792 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.077797 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.077801 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.077805 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.077809 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.077813 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.077817 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.077822 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.077826 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.077830 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.077836 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.077840 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.077845 | controller |       09:26:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.077849 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.077853 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.077857 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.077861 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.077866 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.077870 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.077874 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.077878 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.077882 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.077887 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.077891 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.077895 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.077899 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.077903 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.077907 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.077912 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.077916 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.077926 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.077931 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.077935 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.077939 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.077943 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.077948 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.077956 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.077960 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.077964 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.077968 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.077973 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.077977 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.077983 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.077989 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.077993 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.077997 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.078002 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.078006 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.078010 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.078014 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.078018 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.078023 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.078027 | controller |       09:26:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.078032 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.078038 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.078044 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.078050 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.078056 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.078063 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.078070 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.078077 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.078084 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.078090 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.078097 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.078104 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.078111 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.078118 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.078124 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.078130 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.078135 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.078141 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.078158 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.078164 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.078171 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.078175 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.078179 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.078184 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.078188 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.078192 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.078196 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.078201 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.078205 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.078209 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.078213 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.078217 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.078222 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.078226 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.078230 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.078234 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.078239 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.078243 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.078247 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.078251 | controller |       09:26:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.078255 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.078260 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.078266 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.078271 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.078275 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.078279 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.078283 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.078305 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.078312 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.078316 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.078321 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.078325 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.078329 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.078335 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.078340 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.078344 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.078348 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.078352 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.078356 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.078368 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.078373 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.078377 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.078381 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.078385 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.078390 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.078394 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.078398 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.078402 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.078406 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.078410 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.078415 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.078419 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.078423 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.078427 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.078431 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.078436 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.078440 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.078444 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.078450 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.078456 | controller |       09:26:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.078462 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.078468 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.078474 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.078479 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.078491 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.078496 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.078500 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.078504 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.078508 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.078513 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.078517 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.078521 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.078525 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.078529 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.078534 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.078538 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.078542 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.078546 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.078550 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.078555 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.078568 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.078573 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.078578 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.078582 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.078586 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.078590 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.078594 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.078599 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.078603 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.078607 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.078611 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.078615 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.078620 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.078624 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.078628 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.078632 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.078636 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.078642 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.078646 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.078651 | controller |       09:26:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.078655 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.078659 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.078663 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.078667 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.078672 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.078676 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.078680 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.078684 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.078688 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.078692 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.078697 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.078701 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.078705 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.078709 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.078713 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.078717 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.078722 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.078726 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.078730 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.078734 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.078740 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.078745 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.078755 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.078760 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.078764 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.078768 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.078773 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.078777 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.078781 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.078787 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.078791 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.078795 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.078799 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.078804 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.078808 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.078812 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.078816 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.078820 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.078824 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.078829 | controller |       09:26:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.078833 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.078839 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.078843 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.078847 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.078851 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.078857 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.078861 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.078865 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.078869 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.078874 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.078878 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.078882 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.078886 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.078890 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.078894 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.078899 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.078904 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.078909 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.078913 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.078917 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.078924 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.078928 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.078933 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.078937 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.078948 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.078952 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.078957 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.078961 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.078965 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.078969 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.078974 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.078978 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.078982 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.078986 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.078991 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.078995 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.078999 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.079003 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.079007 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.079011 | controller |       09:26:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.079016 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.079020 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.079024 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.079028 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.079032 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.079037 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.079041 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.079045 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.079049 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.079053 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.079057 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.079061 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.079066 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.079072 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.079076 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.079080 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.079086 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.079090 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.079094 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.079098 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.079103 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.079107 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.079111 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.079115 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.079119 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.079130 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.079135 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.079139 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.079143 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.079147 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.079152 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.079156 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.079160 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.079165 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.079171 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.079176 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.079182 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.079189 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.079195 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.079202 | controller |       09:26:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.079209 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.079215 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.079222 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.079228 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.079238 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.079246 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.079252 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.079257 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.079262 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.079266 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.079270 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.079274 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.079279 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.079283 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.079291 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.079310 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.079314 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.079319 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.079323 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.079327 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.079331 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.079336 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.079340 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.079344 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.079348 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.079352 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.079365 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.079370 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.079374 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.079379 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.079386 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.079390 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.079395 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.079399 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.079403 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.079407 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.079412 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.079418 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.079422 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.079426 | controller |       09:26:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.079432 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.079437 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.079441 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.079446 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.079451 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.079458 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.079464 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.079470 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.079475 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.079479 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.079485 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.079491 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.079496 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.079502 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.079507 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.079511 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.079516 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.079520 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.079524 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.079528 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.079532 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.079537 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.079541 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.079545 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.079549 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.079554 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.079558 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.079562 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.079574 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.079602 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.079608 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.079612 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.079617 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.079621 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.079625 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.079629 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.079634 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.079638 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.079642 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.079646 | controller |       09:26:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.079650 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.079655 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.079659 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.079663 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.079667 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.079671 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.079676 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.079680 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.079684 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.079688 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.079693 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.079697 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.079701 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.079705 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.079712 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.079716 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.079721 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.079725 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.079729 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.079733 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.079741 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.079745 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.079749 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.079753 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.079758 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.079762 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.079766 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.079770 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.079774 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.079785 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.079790 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.079794 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.079798 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.079803 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.079807 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.079811 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.079816 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.079820 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.079824 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.079828 | controller |       09:26:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.079832 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.079837 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.079841 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.079845 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.079851 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.079855 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.079860 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.079864 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.079868 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.079872 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.079877 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.079881 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.079885 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.079891 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.079895 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.079900 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.079904 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.079908 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.079912 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.079917 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.079921 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.079925 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.079930 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.079934 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.079938 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.079942 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.079947 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.079951 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.079955 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.079961 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.079966 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.079977 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.079981 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.079986 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.079990 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.079995 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.079999 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.080003 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.080009 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.080014 | controller |       09:26:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.080019 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.080023 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.080028 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.080032 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.080039 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.080044 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.080048 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.080052 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.080057 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.080061 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.080065 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.080069 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.080074 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.080078 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.080082 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.080086 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.080091 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.080095 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.080099 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.080103 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.080108 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.080112 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.080116 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.080121 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.080125 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.080129 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.080133 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.080138 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.080142 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.080146 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.080150 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.080155 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.080166 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.080170 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.080175 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.080179 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.080183 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.080189 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.080194 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.080198 | controller |       09:26:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.080202 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.080207 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.080211 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.080215 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.080219 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.080224 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.080228 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.080232 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.080237 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.080241 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.080245 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.080249 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.080254 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.080259 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.080264 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.080269 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.080274 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.080278 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.080282 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.080289 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.080319 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.080328 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.080335 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.080343 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.080350 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.080357 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.080364 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.080372 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.080379 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.080388 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.080394 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.080400 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.080406 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.080421 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.080427 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.080432 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.080437 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.080441 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.080446 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.080452 | controller |       09:26:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.080458 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.080464 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.080470 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.080476 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.080483 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.080489 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.080495 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.080501 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.080507 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.080512 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.080517 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.080521 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.080525 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.080529 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.080534 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.080538 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.080542 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.080546 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.080551 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.080555 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.080563 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.080570 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.080576 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.080581 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.080585 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.080589 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.080594 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.080598 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.080602 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.080606 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.080611 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.080615 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.080619 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.080624 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.080640 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.080645 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.080649 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.080654 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.080658 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.080663 | controller |       09:26:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.080667 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.080671 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.080675 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.080680 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.080684 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.080688 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.080692 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.080696 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.080701 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.080705 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.080709 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.080713 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.080717 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.080723 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.080728 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.080732 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.080736 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.080740 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.080744 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.080749 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.080754 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.080759 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.080763 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.080767 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.080771 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.080775 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.080780 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.080784 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.080788 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.080792 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.080798 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.080802 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.080806 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.080810 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.080815 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.080819 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.080830 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.080835 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.080839 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.080843 | controller |       09:26:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.080848 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.080852 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.080856 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.080860 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.080866 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.080871 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.080875 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.080879 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.080883 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.080887 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.080892 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.080896 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.080902 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.080906 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.080910 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.080914 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.080919 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.080923 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.080927 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.080932 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.080938 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.080942 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.080947 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.080952 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.080957 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.080961 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.080966 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.080970 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.080974 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.080978 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.080982 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.080986 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.080991 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.080995 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.080999 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.081003 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.081008 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.081020 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.081024 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.081028 | controller |       09:26:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.081033 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.081037 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.081042 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.081047 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.081051 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.081055 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.081059 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.081064 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.081068 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.081072 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.081076 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.081080 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.081085 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.081089 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.081093 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.081097 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.081101 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.081106 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.081110 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.081114 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.081118 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.081123 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.081127 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.081131 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.081135 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.081139 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.081144 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.081148 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.081152 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.081158 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.081162 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.081166 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.081172 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.081176 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.081180 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.081184 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.081189 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.081193 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.081200 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.081211 | controller |       09:26:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.081216 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.081220 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.081225 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.081229 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.081235 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.081239 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.081243 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.081247 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.081252 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.081256 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.081260 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.081264 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.081268 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.081273 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.081277 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.081281 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.081285 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.081305 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.081313 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.081317 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.081323 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.081328 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.081332 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.081336 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.081340 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.081345 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.081349 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.081353 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.081357 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.081361 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.081366 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.081370 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.081374 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.081378 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.081382 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.081387 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.081391 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.081395 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.081399 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.081404 | controller |       09:26:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.081434 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.081441 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.081446 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.081451 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.081457 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.081462 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.081467 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.081473 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.081478 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.081483 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.081489 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.081497 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.081502 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.081510 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.081515 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.081521 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.081526 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.081531 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.081537 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.081543 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.081549 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.081554 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.081560 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.081565 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.081571 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.081576 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.081582 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.081588 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.081593 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.081599 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.081605 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.081611 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.081616 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.081625 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.081631 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.081636 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.081641 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.081645 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.081649 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.081654 | controller |       09:26:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.081658 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.081674 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.081680 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.081686 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.081694 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.081700 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.081706 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.081712 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.081718 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.081724 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.081730 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.081736 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.081741 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.081746 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.081750 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.081756 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.081761 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.081765 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.081769 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.081773 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.081779 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.081783 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.081788 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.081792 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.081796 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.081800 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.081804 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.081808 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.081814 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.081819 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.081823 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.081827 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.081831 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.081836 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.081840 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.081844 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.081848 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.081854 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.081859 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.081863 | controller |       09:26:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.081867 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.081871 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.081875 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.081888 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.081893 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.081897 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.081901 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.081905 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.081910 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.081914 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.081918 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.081922 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.081926 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.081931 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.081935 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.081939 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.081943 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.081947 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.081952 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.081956 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.081960 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.081964 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.081968 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.081973 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.081977 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.081981 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.081985 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.081989 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.081994 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.081999 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.082004 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.082008 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.082012 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.082016 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.082021 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.082025 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.082029 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.082033 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.082037 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.082042 | controller |       09:26:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.082046 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.082050 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.082054 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.082058 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.082069 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.082074 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.082079 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.082083 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.082087 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.082091 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.082097 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.082102 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.082106 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.082110 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.082114 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.082118 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.082122 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.082127 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.082131 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.082135 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.082141 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.082145 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.082149 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.082154 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.082158 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.082162 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.082166 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.082170 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.082175 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.082179 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.082184 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.082189 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.082193 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.082197 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.082201 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.082206 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.082210 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.082214 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.082218 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.082222 | controller |       09:26:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.082227 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.082231 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.082235 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.082239 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.082243 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.082254 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.082260 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.082264 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.082268 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.082273 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.082277 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.082281 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.082286 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.082311 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.082315 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.082319 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.082324 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.082328 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.082332 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.082336 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.082341 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.082345 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.082349 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.082353 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.082358 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.082362 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.082366 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.082372 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.082376 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.082380 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.082385 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.082389 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.082393 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.082397 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.082402 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.082406 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.082410 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.082414 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.082418 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.082423 | controller |       09:26:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.082427 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.082431 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.082435 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.082451 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.082457 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.082462 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.082466 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.082478 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.082482 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.082487 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.082491 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.082495 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.082500 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.082504 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.082539 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.082544 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.082549 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.082553 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.082557 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.082561 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.082565 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.082570 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.082574 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.082578 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.082582 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.082586 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.082590 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.082595 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.082618 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.082624 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.082628 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.082632 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.082636 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.082640 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.082645 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.082649 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.082653 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.082659 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.082663 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.082668 | controller |       09:26:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.082672 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.082676 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.082680 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.082684 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.082691 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.082695 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.082699 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.082703 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.082715 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.082723 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.082727 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.082731 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.082735 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.082740 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.082744 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.082748 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.082752 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.082756 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.082760 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.082766 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.082774 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.082780 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.082786 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.082790 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.082794 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.082798 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.082803 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.082807 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.082811 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.082817 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.082821 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.082825 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.082830 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.082834 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.082838 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.082842 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.082847 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.082851 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.082855 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.082859 | controller |       09:26:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.082863 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.082868 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.082872 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.082876 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.082880 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.082884 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.082889 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.082893 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.082897 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.082901 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.082913 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.082917 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.082922 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.082926 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.082930 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.082935 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.082939 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.082943 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.082947 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.082952 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.082958 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.082962 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.082966 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.082971 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.082975 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.082981 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.082987 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.082991 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.082995 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.083000 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.083004 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.083008 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.083012 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.083017 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.083021 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.083025 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.083030 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.083034 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.083038 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.083043 | controller |       09:26:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.083047 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.083051 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.083056 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.083060 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.083064 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.083068 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.083073 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.083077 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.083081 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.083086 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.083090 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.083101 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.083105 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.083112 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.083116 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.083122 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.083126 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.083131 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.083135 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.083139 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.083144 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.083148 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.083152 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.083157 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.083161 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.083165 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.083169 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.083174 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.083178 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.083182 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.083187 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.083191 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.083195 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.083200 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.083204 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.083208 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.083213 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.083218 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.083222 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.083226 | controller |       09:26:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.083231 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.083235 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.083239 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.083244 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.083249 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.083278 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.083285 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.083304 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.083314 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.083318 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.083322 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.083327 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.083331 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.083343 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.083348 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.083352 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.083357 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.083361 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.083365 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.083369 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.083374 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.083378 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.083382 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.083386 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.083390 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.083395 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.083399 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.083403 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.083407 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.083412 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.083416 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.083420 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.083424 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.083428 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.083433 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.083437 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.083441 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.083450 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.083455 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.083459 | controller |       09:26:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.083463 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.083467 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.083472 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.083476 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.083480 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.083486 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.083490 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.083494 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.083499 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.083503 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.083507 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.083511 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.083516 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.083520 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.083531 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.083535 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.083540 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.083544 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.083548 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.083553 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.083557 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.083561 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.083565 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.083569 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.083574 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.083578 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.083582 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.083588 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.083592 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.083599 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.083603 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.083607 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.083613 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.083617 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.083621 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.083626 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.083630 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.083634 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.083638 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.083642 | controller |       09:26:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.083647 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.083651 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.083655 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.083659 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.083663 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.083667 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.083672 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.083676 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.083680 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.083684 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.083688 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.083692 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.083696 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.083701 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.083705 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.083715 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.083720 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.083725 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.083729 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.083733 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.083741 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.083745 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.083749 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.083753 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.083758 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.083762 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.083766 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.083770 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.083774 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.083779 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.083783 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.083787 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.083791 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.083796 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.083800 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.083804 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.083808 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.083812 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.083817 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.083821 | controller |       09:26:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.083825 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.083829 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.083835 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.083843 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.083849 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.083855 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.083860 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.083866 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.083872 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.083877 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.083883 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.083889 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.083895 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.083902 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.083906 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.083911 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.083916 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.083931 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.083936 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.083941 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.083946 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.083950 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.083954 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.083958 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.083963 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.083967 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.083971 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.083975 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.083980 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.083984 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.083988 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.083992 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.083997 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.084001 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.084005 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.084009 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.084014 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.084018 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.084022 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.084026 | controller |       09:26:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.084031 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.084035 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.084039 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.084043 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.084052 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.084056 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.084061 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.084065 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.084069 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.084073 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.084077 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.084082 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.084086 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.084090 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.084094 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.084098 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.084103 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.084107 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.084118 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.084122 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.084127 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.084131 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.084135 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.084142 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.084146 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.084150 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.084154 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.084159 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.084163 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.084167 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.084171 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.084175 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.084179 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.084184 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.084188 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.084192 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.084196 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.084202 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.084207 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.084211 | controller |       09:26:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.084215 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.084219 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.084223 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.084228 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.084232 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.084236 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.084240 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.084244 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.084249 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.084253 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.084257 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.084261 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.084265 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.084269 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.084274 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.084278 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.084282 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.084286 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.084306 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.084317 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.084321 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.084326 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.084330 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.084334 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.084338 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.084343 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.084347 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.084351 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.084355 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.084361 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.084365 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.084370 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.084374 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.084378 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.084382 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.084386 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.084390 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.084395 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.084399 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.084403 | controller |       09:26:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.084407 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.084411 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.084418 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.084422 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.084426 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.084430 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.084434 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.084439 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.084443 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.084447 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.084451 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.084455 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.084460 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.084464 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.084468 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.084475 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.084479 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.084484 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.084488 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.084492 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.084504 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.084509 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.084513 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.084518 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.084522 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.084526 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.084530 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.084534 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.084539 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.084543 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.084547 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.084551 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.084555 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.084560 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.084564 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.084568 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.084572 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.084576 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.084581 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.084585 | controller |       09:26:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.084589 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.084593 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.084598 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.084602 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.084606 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.084610 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.084614 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.084619 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.084623 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.084627 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.084631 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.084635 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.084640 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.084645 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.084650 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.084654 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.084658 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.084662 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.084666 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.084671 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.084677 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.084690 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.084701 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.084706 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.084711 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.084715 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.084719 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.084724 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.084728 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.084732 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.084736 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.084740 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.084745 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.084749 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.084753 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.084757 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.084762 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.084766 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.084770 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.084774 | controller |       09:26:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.084779 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.084783 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.084787 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.084791 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.084797 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.084801 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.084805 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.084810 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.084814 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.084818 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.084823 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.084827 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.084831 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.084835 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.084839 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.084844 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.084848 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.084852 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.084856 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.084860 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.084865 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.084869 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.084873 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.084884 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.084888 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.084893 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.084897 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.084901 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.084905 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.084910 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.084916 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.084920 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.084924 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.084928 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.084933 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.084937 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.084941 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.084947 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.084951 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.084955 | controller |       09:26:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.084960 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.084965 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.084970 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.084974 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.084978 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.084982 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.084986 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.084991 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.084995 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.084999 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.085003 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.085008 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.085012 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.085016 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.085020 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.085024 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.085029 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.085033 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.085037 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.085041 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.085046 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.085050 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.085054 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.085059 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.085063 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.085074 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.085078 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.085083 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.085087 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.085093 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.085097 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.085102 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.085106 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.085110 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.085114 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.085119 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.085123 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.085127 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.085132 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.085136 | controller |       09:26:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.085140 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.085144 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.085149 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.085153 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.085157 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.085162 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.085166 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.085170 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.085174 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.085179 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.085183 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.085187 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.085191 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.085196 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.085200 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.085204 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.085210 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.085215 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.085219 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.085223 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.085229 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.085233 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.085238 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.085242 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.085246 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.085250 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.085261 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.085266 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.085270 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.085274 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.085279 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.085283 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.085289 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.085307 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.085312 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.085316 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.085320 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.085324 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.085328 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.085333 | controller |       09:26:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.085337 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.085341 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.085345 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.085350 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.085356 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.085360 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.085365 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.085369 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.085373 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.085377 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.085382 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.085386 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.085390 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.085396 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.085400 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.085404 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.085408 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.085413 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.085417 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.085422 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.085442 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.085448 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.085453 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.085458 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.085463 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.085468 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.085473 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.085477 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.085490 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.085496 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.085501 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.085506 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.085511 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.085516 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.085521 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.085526 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.085531 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.085536 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.085541 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.085546 | controller |       09:26:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.085553 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.085558 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.085563 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.085568 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.085576 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.085581 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.085586 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.085591 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.085596 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.085601 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.085606 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.085610 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.085615 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.085620 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.085625 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.085630 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.085635 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.085640 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.085645 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.085650 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.085657 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.085662 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.085667 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.085671 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.085676 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.085681 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.085686 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.085690 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.085694 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.085705 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.085710 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.085714 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.085718 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.085723 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.085727 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.085731 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.085735 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.085741 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.085748 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.085752 | controller |       09:26:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.085756 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.085760 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.085765 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.085769 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.085773 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.085777 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.085782 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.085786 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.085790 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.085794 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.085798 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.085803 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.085807 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.085811 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.085817 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.085822 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.085826 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.085830 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.085835 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.085839 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.085843 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.085847 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.085851 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.085856 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.085860 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.085864 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.085868 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.085872 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.085877 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.085882 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.085893 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.085898 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.085902 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.085906 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.085911 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.085915 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.085919 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.085923 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.085927 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.085932 | controller |       09:26:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.085936 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.085940 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.085945 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.085949 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.085953 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.085957 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.085962 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.085966 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.085970 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.085974 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.085979 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.085983 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.085988 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.085992 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.085997 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.086001 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.086005 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.086010 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.086014 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.086018 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.086024 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.086028 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.086033 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.086037 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.086042 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.086048 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.086054 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.086060 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.086065 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.086071 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.086077 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.086083 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.086095 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.086100 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.086105 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.086110 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.086114 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.086118 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.086125 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.086129 | controller |       09:26:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.086134 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.086138 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.086142 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.086147 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.086151 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.086155 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.086159 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.086164 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.086168 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.086172 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.086176 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.086181 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.086185 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.086191 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.086195 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.086200 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.086204 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.086208 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.086212 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.086217 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.086221 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.086225 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.086229 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.086234 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.086238 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.086242 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.086246 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.086251 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.086255 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.086259 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.086265 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.086269 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.086274 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.086285 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.086309 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.086316 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.086320 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.086324 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.086328 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.086333 | controller |       09:26:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.086337 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:26:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.086341 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.086345 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.086349 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.086355 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.086360 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.086364 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.086368 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.086372 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.086376 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.086381 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.086385 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.086389 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.086395 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.086399 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.086404 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.086408 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.086412 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.086416 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.086420 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.086424 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.086429 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.086433 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.086437 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.086441 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.086445 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.086450 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.086454 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.086458 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.086462 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.086466 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.086470 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.086475 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.086479 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.086491 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.086495 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.086499 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.086506 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:26:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.086510 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.086515 | controller |       09:26:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.086519 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.086523 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.086527 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.086532 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.086536 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.086540 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.086544 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.086548 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.086553 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.086557 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.086561 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.086565 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.086569 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.086574 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.086578 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.086582 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.086586 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.086590 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.086595 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.086599 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.086605 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.086609 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.086613 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.086617 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.086621 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.086627 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.086631 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.086635 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.086639 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.086647 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.086651 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.086655 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.086660 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.086664 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.086668 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.086674 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.086685 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.086690 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.086694 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.086698 | controller |       09:27:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.086703 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.086707 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.086711 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.086715 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.086721 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.086726 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.086730 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.086734 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.086738 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.086742 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.086747 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.086751 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.086755 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.086759 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.086763 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.086768 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.086772 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.086776 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.086780 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.086784 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.086790 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.086794 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.086799 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.086803 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.086807 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.086811 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.086815 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.086820 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.086824 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.086828 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.086832 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.086837 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.086841 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.086845 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.086849 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.086854 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.086858 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.086868 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.086873 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.086877 | controller |       09:27:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.086882 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.086886 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.086890 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.086894 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.086898 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.086903 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.086907 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.086911 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.086915 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.086919 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.086924 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.086928 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.086933 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.086940 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.086944 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.086948 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.086952 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.086956 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.086961 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.086965 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.086969 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.086973 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.086977 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.086982 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.086986 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.086990 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.086994 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.086998 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.087003 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.087007 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.087011 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.087015 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.087019 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.087024 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.087028 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.087032 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.087036 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.087040 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.087050 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.087055 | controller |       09:27:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.087059 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.087064 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.087068 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.087072 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.087078 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.087082 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.087087 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.087091 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.087095 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.087099 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.087103 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.087108 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.087112 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.087116 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.087120 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.087126 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.087130 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.087134 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.087139 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.087143 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.087147 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.087152 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.087157 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.087161 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.087165 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.087169 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.087173 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.087178 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.087182 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.087186 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.087190 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.087194 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.087200 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.087204 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.087209 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.087213 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.087217 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.087223 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.087228 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.087232 | controller |       09:27:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.087244 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.087248 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.087253 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.087257 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.087261 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.087265 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.087269 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.087274 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.087278 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.087282 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.087288 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.087304 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.087310 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.087314 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.087318 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.087322 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.087327 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.087331 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.087335 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.087339 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.087344 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.087348 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.087352 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.087357 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.087361 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.087365 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.087369 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.087374 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.087378 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.087384 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.087389 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.087393 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.087397 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.087402 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.087406 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.087410 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.087415 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.087419 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.087423 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.087427 | controller |       09:27:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.087432 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.087442 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.087446 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.087451 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.087455 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.087459 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.087464 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.087468 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.087472 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.087476 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.087481 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.087487 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.087492 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.087496 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.087501 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.087505 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.087509 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.087513 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.087518 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.087522 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.087533 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.087537 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.087542 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.087546 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.087550 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.087554 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.087559 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.087564 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.087568 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.087572 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.087578 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.087582 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.087587 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.087594 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.087598 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.087603 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.087607 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.087611 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.087616 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.087620 | controller |       09:27:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.087624 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.087628 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.087638 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.087643 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.087647 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.087652 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.087656 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.087660 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.087664 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.087669 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.087673 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.087677 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.087682 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.087688 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.087692 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.087696 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.087701 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.087705 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.087709 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.087713 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.087718 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.087722 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.087726 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.087730 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.087735 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.087739 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.087743 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.087748 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.087754 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.087758 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.087762 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.087766 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.087771 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.087775 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.087779 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.087784 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.087788 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.087792 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.087796 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.087801 | controller |       09:27:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.087805 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.087809 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.087813 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.087818 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.087829 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.087835 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.087840 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.087844 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.087848 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.087853 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.087857 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.087861 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.087865 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.087870 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.087874 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.087878 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.087883 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.087887 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.087891 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.087895 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.087900 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.087904 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.087908 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.087913 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.087918 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.087924 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.087928 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.087933 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.087938 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.087943 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.087947 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.087952 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.087956 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.087960 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.087965 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.087969 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.087973 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.087979 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.087983 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.087988 | controller |       09:27:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.087992 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.087996 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.088000 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.088005 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.088010 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.088021 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.088026 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.088031 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.088035 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.088039 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.088046 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.088050 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.088054 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.088059 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.088063 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.088067 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.088071 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.088076 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.088080 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.088084 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.088088 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.088093 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.088097 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.088101 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.088106 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.088110 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.088114 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.088118 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.088123 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.088128 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.088133 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.088137 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.088141 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.088146 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.088150 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.088154 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.088158 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.088163 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.088167 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.088171 | controller |       09:27:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.088176 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.088180 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.088184 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.088188 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.088193 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.088197 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.088201 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.088211 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.088216 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.088221 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.088227 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.088233 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.088239 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.088245 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.088249 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.088253 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.088258 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.088262 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.088266 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.088270 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.088278 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.088282 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.088289 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.088307 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.088311 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.088316 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.088320 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.088327 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.088331 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.088336 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.088340 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.088344 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.088348 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.088353 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.088357 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.088361 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.088366 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.088370 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.088374 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.088379 | controller |       09:27:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.088383 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.088387 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.088392 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.088396 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.088400 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.088404 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.088409 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.088413 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.088424 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.088428 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.088433 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.088437 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.088441 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.088448 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.088453 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.088458 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.088463 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.088467 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.088471 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.088476 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.088481 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.088486 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.088491 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.088495 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.088499 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.088504 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.088508 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.088512 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.088516 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.088521 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.088525 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.088529 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.088534 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.088538 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.088542 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.088546 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.088551 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.088555 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.088559 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.088564 | controller |       09:27:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.088568 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.088572 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.088577 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.088581 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.088587 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.088591 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.088595 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.088600 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.088604 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.088617 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.088623 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.088628 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.088632 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.088637 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.088641 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.088645 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.088649 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.088654 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.088658 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.088662 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.088667 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.088671 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.088675 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.088680 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.088684 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.088688 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.088692 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.088697 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.088701 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.088705 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.088710 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.088714 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.088718 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.088722 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.088727 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.088731 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.088735 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.088741 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.088746 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.088750 | controller |       09:27:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.088754 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.088758 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.088763 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.088767 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.088771 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.088775 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.088779 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.088784 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.088788 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.088792 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.088796 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.088806 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.088811 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.088815 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.088819 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.088824 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.088828 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.088832 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.088836 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.088841 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.088846 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.088850 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.088854 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.088858 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.088863 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.088869 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.088874 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.088878 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.088882 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.088888 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.088894 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.088898 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.088903 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.088907 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.088912 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.088916 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.088920 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.088924 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.088928 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.088933 | controller |       09:27:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.088937 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.088941 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.088945 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.088949 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.088954 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.088958 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.088962 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.088966 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.088970 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.088975 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.088979 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.088983 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.088993 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.088997 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.089002 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.089006 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.089010 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.089015 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.089019 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.089023 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.089029 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.089033 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.089038 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.089042 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.089046 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.089050 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.089054 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.089059 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.089063 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.089067 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.089071 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.089076 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.089080 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.089084 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.089088 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.089092 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.089097 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.089101 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.089105 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.089109 | controller |       09:27:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.089114 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.089118 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.089122 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.089127 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.089131 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.089135 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.089139 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.089144 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.089150 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.089154 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.089159 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.089163 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.089167 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.089173 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.089183 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.089188 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.089192 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.089196 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.089201 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.089205 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.089209 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.089214 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.089218 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.089222 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.089226 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.089231 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.089236 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.089240 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.089244 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.089249 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.089253 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.089257 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.089261 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.089266 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.089270 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.089274 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.089279 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.089283 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.089302 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.089313 | controller |       09:27:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.089320 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.089326 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.089332 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.089338 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.089349 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.089355 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.089361 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.089366 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.089372 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.089378 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.089382 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.089386 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.089391 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.089395 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.089399 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.089413 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.089418 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.089422 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.089427 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.089431 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.089452 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.089457 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.089461 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.089467 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.089474 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.089480 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.089486 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.089492 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.089498 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.089503 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.089509 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.089514 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.089520 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.089524 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.089528 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.089533 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.089537 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.089544 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.089549 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.089553 | controller |       09:27:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.089558 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.089562 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.089567 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.089571 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.089575 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.089579 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.089584 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.089588 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.089592 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.089596 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.089601 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.089605 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.089609 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.089613 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.089618 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.089622 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.089626 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.089639 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.089643 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.089647 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.089652 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.089656 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.089660 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.089665 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.089669 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.089673 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.089677 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.089681 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.089686 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.089692 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.089696 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.089700 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.089704 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.089709 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.089713 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.089717 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.089721 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.089725 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.089730 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.089734 | controller |       09:27:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.089738 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.089742 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.089747 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.089753 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.089757 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.089761 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.089766 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.089770 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.089774 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.089778 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.089782 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.089787 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.089791 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.089795 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.089799 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.089806 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.089811 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.089815 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.089825 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.089830 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.089836 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.089840 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.089845 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.089849 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.089853 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.089857 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.089862 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.089866 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.089870 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.089874 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.089878 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.089883 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.089887 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.089891 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.089895 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.089899 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.089904 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.089908 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.089912 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.089916 | controller |       09:27:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.089920 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.089925 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.089929 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.089933 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.089937 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.089942 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.089946 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.089950 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.089954 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.089958 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.089963 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.089967 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.089971 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.089977 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.089981 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.089985 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.089989 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.089994 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.089999 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.090009 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.090014 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.090018 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.090022 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.090028 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.090033 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.090037 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.090041 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.090045 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.090049 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.090054 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.090058 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.090062 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.090066 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.090073 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.090077 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.090081 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.090085 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.090090 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.090095 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.090099 | controller |       09:27:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.090104 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.090108 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.090112 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.090116 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.090122 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.090126 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.090131 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.090135 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.090139 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.090143 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.090147 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.090151 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.090156 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.090160 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.090164 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.090168 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.090172 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.090177 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.090181 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.090185 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.090196 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.090200 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.090204 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.090209 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.090213 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.090217 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.090222 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.090226 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.090230 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.090234 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.090240 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.090246 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.090250 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.090254 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.090258 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.090263 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.090267 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.090273 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.090277 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.090281 | controller |       09:27:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.090285 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.090303 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.090312 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.090316 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.090320 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.090325 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.090329 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.090333 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.090337 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.090342 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.090346 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.090350 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.090354 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.090358 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.090363 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.090367 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.090371 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.090375 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.090379 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.090384 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.090389 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.090393 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.090405 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.090410 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.090414 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.090419 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.090423 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.090427 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.090431 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.090437 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.090442 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.090446 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.090450 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.090454 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.090458 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.090463 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.090467 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.090471 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.090475 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.090479 | controller |       09:27:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.090484 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.090488 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.090492 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.090496 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.090500 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.090505 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.090509 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.090513 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.090517 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.090521 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.090526 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.090530 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.090534 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.090538 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.090542 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.090547 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.090551 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.090555 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.090559 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.090563 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.090569 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.090575 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.090579 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.090590 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.090594 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.090598 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.090603 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.090607 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.090611 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.090616 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.090620 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.090624 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.090628 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.090633 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.090637 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.090641 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.090645 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.090649 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.090654 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.090658 | controller |       09:27:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.090662 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.090666 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.090671 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.090675 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.090682 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.090686 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.090690 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.090694 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.090699 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.090703 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.090707 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.090711 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.090715 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.090721 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.090725 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.090730 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.090734 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.090738 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.090742 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.090746 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.090751 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.090755 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.090759 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.090763 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.090768 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.090778 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.090782 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.090787 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.090791 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.090795 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.090799 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.090804 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.090808 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.090812 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.090816 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.090821 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.090825 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.090829 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.090833 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.090837 | controller |       09:27:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.090842 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.090848 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.090852 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.090856 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.090862 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.090866 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.090871 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.090875 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.090879 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.090883 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.090887 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.090892 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.090896 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.090900 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.090904 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.090908 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.090913 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.090917 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.090921 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.090926 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.090930 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.090934 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.090938 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.090942 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.090947 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.090951 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.090961 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.090966 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.090970 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.090974 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.090979 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.090983 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.090987 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.090991 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.090995 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.091000 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.091004 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.091010 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.091014 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.091018 | controller |       09:27:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.091022 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.091027 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.091031 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.091035 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.091039 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.091044 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.091048 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.091052 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.091056 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.091061 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.091066 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.091070 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.091074 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.091078 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.091082 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.091088 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.091094 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.091098 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.091102 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.091107 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.091111 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.091115 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.091119 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.091123 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.091128 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.091132 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.091136 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.091140 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.091150 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.091156 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.091161 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.091165 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.091169 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.091173 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.091177 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.091182 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.091186 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.091190 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.091194 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.091198 | controller |       09:27:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.091203 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.091207 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.091211 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.091215 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.091219 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.091224 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.091228 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.091232 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.091236 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.091240 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.091245 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.091249 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.091253 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.091257 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.091261 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.091265 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.091270 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.091274 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.091278 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.091282 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.091303 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.091310 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.091314 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.091318 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.091322 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.091327 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.091331 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.091335 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.091339 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.091349 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.091354 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.091358 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.091362 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.091367 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.091371 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.091375 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.091379 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.091383 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.091388 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.091392 | controller |       09:27:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.091398 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.091402 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.091406 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.091411 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.091415 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.091419 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.091423 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.091428 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.091432 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.091436 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.091440 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.091444 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.091449 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.091455 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.091459 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.091463 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.091467 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.091472 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.091477 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.091483 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.091489 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.091495 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.091500 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.091504 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.091508 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.091512 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.091516 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.091521 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.091525 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.091529 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.091541 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.091546 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.091551 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.091555 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.091559 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.091563 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.091568 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.091572 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.091576 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.091580 | controller |       09:27:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.091585 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.091589 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.091593 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.091597 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.091604 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.091608 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.091612 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.091616 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.091620 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.091625 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.091629 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.091633 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.091637 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.091642 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.091647 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.091651 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.091656 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.091660 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.091664 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.091668 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.091673 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.091677 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.091681 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.091685 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.091689 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.091694 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.091698 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.091702 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.091706 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.091710 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.091715 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.091719 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.091728 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.091733 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.091737 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.091742 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.091746 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.091753 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.091758 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.091762 | controller |       09:27:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.091766 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.091770 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.091774 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.091779 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.091783 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.091787 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.091791 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.091795 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.091800 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.091804 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.091808 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.091812 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.091816 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.091821 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.091825 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.091829 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.091833 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.091837 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.091842 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.091846 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.091850 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.091854 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.091858 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.091863 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.091867 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.091871 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.091875 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.091879 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.091883 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.091889 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.091893 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.091898 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.091902 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.091911 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.091916 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.091920 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.091924 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.091929 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.091935 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.091939 | controller |       09:27:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.091943 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.091947 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.091952 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.091956 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.091961 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.091966 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.091970 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.091974 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.091978 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.091983 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.091987 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.091991 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.091995 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.091999 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.092004 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.092008 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.092012 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.092016 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.092020 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.092025 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.092030 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.092035 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.092039 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.092043 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.092047 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.092051 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.092056 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.092060 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.092064 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.092068 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.092072 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.092076 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.092081 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.092085 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.092095 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.092099 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.092103 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.092108 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.092112 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.092116 | controller |       09:27:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.092121 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.092125 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.092129 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.092133 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.092137 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.092143 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.092147 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.092152 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.092156 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.092160 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.092164 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.092168 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.092173 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.092178 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.092183 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.092187 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.092193 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.092197 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.092201 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.092205 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.092210 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.092215 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.092219 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.092224 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.092228 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.092232 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.092236 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.092240 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.092245 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.092249 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.092253 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.092257 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.092261 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.092266 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.092270 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.092274 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.092283 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.092301 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.092307 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.092312 | controller |       09:27:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.092316 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.092320 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.092324 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.092328 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.092334 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.092339 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.092343 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.092347 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.092351 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.092356 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.092360 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.092364 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.092368 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.092372 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.092377 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.092382 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.092387 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.092391 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.092395 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.092399 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.092404 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.092408 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.092412 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.092416 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.092420 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.092425 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.092429 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.092433 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.092437 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.092441 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.092446 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.092450 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.092454 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.092458 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.092462 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.092468 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.092472 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.092485 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.092490 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.092495 | controller |       09:27:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.092499 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.092503 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.092507 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.092512 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.092516 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.092520 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.092524 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.092528 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.092533 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.092537 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.092541 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.092545 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.092549 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.092554 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.092560 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.092566 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.092572 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.092577 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.092581 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.092585 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.092589 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.092594 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.092598 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.092602 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.092606 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.092610 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.092615 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.092619 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.092623 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.092629 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.092633 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.092638 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.092642 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.092646 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.092650 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.092654 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.092659 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.092663 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.092673 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.092678 | controller |       09:27:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.092682 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.092687 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.092691 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.092695 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.092699 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.092704 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.092708 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.092712 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.092716 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.092720 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.092724 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.092729 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.092735 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.092739 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.092743 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.092747 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.092752 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.092759 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.092763 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.092768 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.092774 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.092778 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.092782 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.092786 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.092790 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.092795 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.092799 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.092803 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.092807 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.092811 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.092818 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.092822 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.092827 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.092831 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.092835 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.092839 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.092844 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.092848 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.092852 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.092862 | controller |       09:27:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.092867 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.092872 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.092876 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.092880 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.092884 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.092888 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.092893 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.092897 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.092901 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.092905 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.092909 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.092914 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.092918 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.092924 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.092928 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.092932 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.092936 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.092941 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.092945 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.092949 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.092953 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.092958 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.092962 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.092966 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.092970 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.092974 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.092979 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.092983 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.092987 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.092991 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.092995 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.093000 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.093005 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.093010 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.093014 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.093018 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.093022 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.093027 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.093031 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.093035 | controller |       09:27:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.093039 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.093049 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.093053 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.093058 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.093064 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.093069 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.093073 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.093077 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.093081 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.093086 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.093090 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.093094 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.093098 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.093102 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.093107 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.093111 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.093115 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.093119 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.093124 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.093128 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.093134 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.093138 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.093142 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.093146 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.093150 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.093155 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.093159 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.093163 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.093167 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.093171 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.093176 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.093180 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.093184 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.093188 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.093193 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.093197 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.093201 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.093207 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.093211 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.093215 | controller |       09:27:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.093220 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.093224 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.093234 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.093238 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.093244 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.093249 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.093253 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.093257 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.093262 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.093266 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.093270 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.093283 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.093290 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.093310 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.093314 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.093318 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.093323 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.093327 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.093331 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.093335 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.093340 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.093344 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.093348 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.093352 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.093356 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.093361 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.093365 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.093369 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.093373 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.093379 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.093383 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.093388 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.093392 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.093396 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.093400 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.093404 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.093409 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.093413 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.093417 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.093421 | controller |       09:27:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.093426 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.093430 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.093435 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.093440 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.093468 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.093473 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.093478 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.093482 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.093486 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.093490 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.093495 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.093499 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.093503 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.093507 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.093511 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.093516 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.093520 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.093524 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.093528 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.093532 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.093539 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.093543 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.093547 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.093551 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.093556 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.093560 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.093564 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.093568 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.093574 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.093579 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.093583 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.093587 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.093591 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.093596 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.093600 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.093604 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.093608 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.093612 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.093617 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.093621 | controller |       09:27:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.093625 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.093629 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.093633 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.093638 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.093642 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.093651 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.093656 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.093660 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.093664 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.093669 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.093673 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.093677 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.093681 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.093687 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.093692 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.093697 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.093701 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.093706 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.093710 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.093714 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.093718 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.093722 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.093727 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.093731 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.093735 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.093739 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.093743 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.093748 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.093752 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.093756 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.093760 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.093764 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.093769 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.093773 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.093777 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.093781 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.093785 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.093790 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.093794 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.093798 | controller |       09:27:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.093802 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.093806 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.093811 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.093815 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.093822 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.093826 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.093836 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.093841 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.093845 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.093850 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.093856 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.093860 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.093864 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.093868 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.093872 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.093877 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.093881 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.093885 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.093889 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.093893 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.093898 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.093902 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.093906 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.093910 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.093916 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.093921 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.093925 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.093930 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.093935 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.093940 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.093945 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.093949 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.093953 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.093957 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.093961 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.093965 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.093970 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.093975 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.093980 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.093984 | controller |       09:27:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.093988 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.093992 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.093997 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.094001 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.094005 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.094009 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.094013 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.094018 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.094027 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.094031 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.094036 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.094040 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.094044 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.094048 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.094053 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.094057 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.094061 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.094065 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.094069 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.094074 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.094079 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.094084 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.094088 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.094092 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.094096 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.094100 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.094104 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.094110 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.094114 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.094120 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.094126 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.094130 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.094134 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.094138 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.094143 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.094147 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.094151 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.094155 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.094159 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.094164 | controller |       09:27:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.094168 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.094172 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.094176 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.094180 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.094185 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.094189 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.094193 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.094197 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.094201 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.094211 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.094216 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.094220 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.094224 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.094229 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.094233 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.094237 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.094241 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.094246 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.094250 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.094254 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.094260 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.094264 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.094268 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.094272 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.094276 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.094281 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.094285 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.094302 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.094309 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.094313 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.094317 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.094322 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.094326 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.094330 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.094334 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.094338 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.094343 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.094347 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.094353 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.094357 | controller |       09:27:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.094361 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.094366 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.094370 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.094374 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.094378 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.094382 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.094387 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.094391 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.094395 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.094401 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.094405 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.094416 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.094420 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.094426 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.094431 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.094435 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.094439 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.094443 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.094447 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.094452 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.094456 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.094460 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.094464 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.094469 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.094473 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.094477 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.094481 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.094485 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.094490 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.094494 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.094498 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.094502 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.094507 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.094511 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.094515 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.094519 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.094523 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.094528 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.094532 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.094536 | controller |       09:27:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.094540 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.094545 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.094549 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.094553 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.094560 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.094564 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.094569 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.094573 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.094577 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.094581 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.094585 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.094590 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.094599 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.094604 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.094608 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.094612 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.094617 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.094621 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.094625 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.094629 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.094633 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.094638 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.094642 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.094646 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.094650 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.094656 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.094660 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.094664 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.094669 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.094673 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.094677 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.094681 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.094685 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.094690 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.094694 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.094698 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.094702 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.094708 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.094712 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.094716 | controller |       09:27:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.094721 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.094727 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.094733 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.094739 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.094744 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.094748 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.094752 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.094757 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.094761 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.094765 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.094769 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.094773 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.094778 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.094782 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.094792 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.094797 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.094802 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.094806 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.094810 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.094815 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.094819 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.094823 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.094827 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.094831 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.094836 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.094840 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.094844 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.094848 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.094852 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.094860 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.094864 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.094868 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.094872 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.094877 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.094881 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.094885 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.094889 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.094893 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.094898 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.094902 | controller |       09:27:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.094906 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.094910 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.094914 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.094919 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.094923 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.094927 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.094931 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.094936 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.094942 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.094946 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.094950 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.094954 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.094959 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.094963 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.094967 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.094978 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.094983 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.094987 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.094991 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.094995 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.095004 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.095008 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.095012 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.095016 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.095021 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.095025 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.095029 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.095033 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.095037 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.095042 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.095046 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.095050 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.095054 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.095059 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.095063 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.095067 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.095071 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.095075 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.095080 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.095084 | controller |       09:27:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.095088 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.095092 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.095097 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.095101 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.095105 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.095109 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.095113 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.095118 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.095122 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.095126 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.095130 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.095134 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.095139 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.095144 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.095149 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.095153 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.095157 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.095167 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.095171 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.095176 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.095180 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.095184 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.095189 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.095193 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.095199 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.095203 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.095208 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.095212 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.095216 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.095220 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.095224 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.095229 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.095233 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.095237 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.095241 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.095245 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.095250 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.095254 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.095258 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.095262 | controller |       09:27:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.095266 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.095271 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.095275 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.095279 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.095285 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.095302 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.095310 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.095314 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.095318 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.095322 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.095327 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.095331 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.095335 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.095339 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.095344 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.095348 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.095352 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.095356 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.095367 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.095371 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.095376 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.095380 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.095384 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.095388 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.095393 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.095397 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.095401 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.095405 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.095410 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.095414 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.095420 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.095424 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.095428 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.095432 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.095437 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.095441 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.095445 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.095451 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.095455 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.095460 | controller |       09:27:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.095464 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.095468 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.095472 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.095478 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.095482 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.095487 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.095491 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.095495 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.095499 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.095503 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.095508 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.095512 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.095516 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.095520 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.095524 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.095529 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.095533 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.095537 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.095541 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.095551 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.095555 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.095560 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.095564 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.095568 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.095572 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.095577 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.095581 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.095585 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.095589 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.095595 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.095599 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.095603 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.095608 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.095612 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.095616 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.095620 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.095624 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.095629 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.095633 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.095637 | controller |       09:27:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.095641 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.095645 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.095650 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.095654 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.095658 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.095662 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.095666 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.095671 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.095675 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.095679 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.095683 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.095687 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.095692 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.095696 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.095700 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.095704 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.095708 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.095713 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.095717 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.095721 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.095732 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.095737 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.095741 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.095747 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.095752 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.095756 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.095760 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.095764 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.095769 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.095773 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.095777 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.095781 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.095785 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.095790 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.095794 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.095798 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.095802 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.095809 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.095813 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.095817 | controller |       09:27:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.095822 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.095826 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.095830 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.095834 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.095841 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.095845 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.095850 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.095854 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.095858 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.095862 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.095866 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.095871 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.095875 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.095881 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.095885 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.095889 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.095893 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.095898 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.095902 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.095906 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.095912 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.095916 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.095926 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.095931 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.095935 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.095940 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.095944 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.095948 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.095952 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.095956 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.095961 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.095965 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.095969 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.095973 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.095977 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.095982 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.095986 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.095990 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.095994 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.095998 | controller |       09:27:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.096003 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.096007 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.096013 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.096017 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.096023 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.096027 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.096031 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.096035 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.096039 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.096044 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.096048 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.096052 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.096056 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.096060 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.096065 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.096069 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.096073 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.096077 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.096081 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.096086 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.096090 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.096094 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.096098 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.096108 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.096112 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.096117 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.096121 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.096125 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.096130 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.096134 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.096138 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.096142 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.096147 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.096151 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.096155 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.096159 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.096163 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.096169 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.096173 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.096178 | controller |       09:27:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.096182 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:27:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.096186 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.096190 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.096195 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.096199 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.096203 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.096207 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.096211 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.096216 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.096220 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.096224 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.096228 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.096232 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.096237 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.096241 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.096246 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.096251 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.096255 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.096259 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.096263 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.096268 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.096273 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.096278 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.096282 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.096303 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.096310 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.096315 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.096319 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.096323 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.096329 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.096333 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.096338 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.096342 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.096346 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.096350 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.096355 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.096359 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.096363 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:27:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.096367 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.096372 | controller |       09:27:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.096376 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.096380 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.096384 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.096388 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.096393 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.096399 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.096403 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.096407 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.096411 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.096416 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.096420 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.096424 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.096428 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.096432 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.096437 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.096441 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.096445 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.096449 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.096454 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.096458 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.096464 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.096468 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.096472 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.096476 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.096480 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.096485 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.096495 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.096499 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.096504 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.096508 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.096512 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.096517 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.096521 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.096525 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.096529 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.096534 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.096538 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.096542 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.096546 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.096550 | controller |       09:28:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.096555 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.096560 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.096565 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.096569 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.096573 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.096577 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.096581 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.096586 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.096590 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.096594 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.096598 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.096602 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.096607 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.096612 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.096617 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.096621 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.096625 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.096629 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.096634 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.096638 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.096642 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.096646 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.096650 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.096655 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.096659 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.096663 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.096667 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.096677 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.096681 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.096685 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.096691 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.096695 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.096700 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.096704 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.096708 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.096712 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.096717 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.096721 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.096725 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.096729 | controller |       09:28:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.096734 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.096738 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.096742 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.096746 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.096752 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.096756 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.096761 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.096765 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.096769 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.096773 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.096777 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.096782 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.096786 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.096790 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.096794 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.096798 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.096805 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.096810 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.096814 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.096818 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.096824 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.096828 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.096832 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.096836 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.096840 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.096845 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.096849 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.096853 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.096857 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.096867 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.096873 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.096877 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.096881 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.096886 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.096892 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.096898 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.096904 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.096911 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.096915 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.096919 | controller |       09:28:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.096923 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.096928 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.096932 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.096936 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.096940 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.096944 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.096949 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.096953 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.096957 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.096961 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.096966 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.096970 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.096974 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.096978 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.096982 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.096987 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.096991 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.096995 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.096999 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.097003 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.097008 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.097012 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.097016 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.097020 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.097024 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.097029 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.097033 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.097037 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.097041 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.097048 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.097058 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.097063 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.097067 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.097072 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.097076 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.097080 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.097084 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.097089 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.097093 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.097097 | controller |       09:28:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.097103 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.097107 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.097112 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.097116 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.097121 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.097126 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.097130 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.097134 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.097138 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.097143 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.097147 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.097151 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.097155 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.097159 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.097164 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.097168 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.097172 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.097176 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.097180 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.097185 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.097190 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.097195 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.097199 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.097203 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.097207 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.097212 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.097216 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.097220 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.097224 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.097228 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.097232 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.097242 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.097246 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.097251 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.097255 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.097259 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.097264 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.097269 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.097274 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.097280 | controller |       09:28:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.097286 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.097310 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.097316 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.097321 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.097325 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.097329 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.097333 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.097337 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.097342 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.097346 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.097350 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.097354 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.097358 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.097365 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.097371 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.097376 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.097380 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.097384 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.097388 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.097392 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.097397 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.097401 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.097405 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.097409 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.097413 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.097418 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.097422 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.097426 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.097430 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.097434 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.097439 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.097443 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.097448 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.097485 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.097491 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.097497 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.097502 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.097508 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.097513 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.097518 | controller |       09:28:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.097524 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.097529 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.097535 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.097540 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.097549 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.097554 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.097560 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.097566 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.097571 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.097576 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.097581 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.097585 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.097589 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.097593 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.097597 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.097604 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.097608 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.097612 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.097617 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.097621 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.097625 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.097629 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.097634 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.097638 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.097642 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.097647 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.097651 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.097655 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.097660 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.097664 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.097668 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.097672 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.097676 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.097681 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.097692 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.097697 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.097701 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.097707 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.097713 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.097718 | controller |       09:28:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.097722 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.097726 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.097730 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.097734 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.097739 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.097743 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.097747 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.097751 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.097755 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.097760 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.097764 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.097768 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.097772 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.097776 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.097780 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.097785 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.097789 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.097793 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.097797 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.097801 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.097807 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.097811 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.097816 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.097820 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.097824 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.097828 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.097832 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.097837 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.097841 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.097847 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.097851 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.097855 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.097859 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.097863 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.097868 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.097879 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.097883 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.097887 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.097892 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.097896 | controller |       09:28:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.097900 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.097904 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.097909 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.097913 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.097917 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.097921 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.097925 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.097930 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.097934 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.097938 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.097942 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.097946 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.097951 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.097958 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.097962 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.097967 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.097973 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.097979 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.097985 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.097990 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.097996 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.098000 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.098004 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.098008 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.098013 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.098017 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.098021 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.098025 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.098029 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.098034 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.098039 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.098044 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.098048 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.098052 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.098056 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.098060 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.098065 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.098076 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.098081 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.098085 | controller |       09:28:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.098089 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.098094 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.098098 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.098102 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.098106 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.098110 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.098115 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.098119 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.098123 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.098127 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.098131 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.098135 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.098140 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.098145 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.098150 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.098154 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.098158 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.098162 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.098166 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.098171 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.098175 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.098179 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.098183 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.098187 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.098192 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.098196 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.098200 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.098204 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.098208 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.098212 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.098217 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.098221 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.098225 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.098230 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.098235 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.098240 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.098245 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.098249 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.098260 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.098264 | controller |       09:28:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.098269 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.098273 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.098277 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.098282 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.098299 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.098307 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.098311 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.098316 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.098320 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.098324 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.098328 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.098333 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.098337 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.098341 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.098345 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.098349 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.098354 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.098358 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.098362 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.098366 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.098371 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.098375 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.098379 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.098383 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.098388 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.098392 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.098396 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.098400 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.098404 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.098409 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.098413 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.098417 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.098421 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.098426 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.098430 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.098434 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.098438 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.098444 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.098448 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.098461 | controller |       09:28:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.098466 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.098470 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.098475 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.098479 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.098483 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.098487 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.098492 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.098496 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.098500 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.098504 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.098509 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.098513 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.098519 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.098523 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.098527 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.098531 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.098535 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.098540 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.098544 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.098548 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.098552 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.098556 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.098561 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.098565 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.098569 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.098573 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.098577 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.098582 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.098586 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.098592 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.098596 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.098600 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.098604 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.098609 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.098613 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.098617 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.098621 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.098625 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.098631 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.098635 | controller |       09:28:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.098640 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.098650 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.098655 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.098659 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.098663 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.098668 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.098672 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.098676 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.098680 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.098684 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.098689 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.098693 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.098697 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.098701 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.098705 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.098710 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.098714 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.098718 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.098722 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.098726 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.098734 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.098738 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.098742 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.098746 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.098751 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.098755 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.098759 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.098763 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.098767 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.098771 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.098776 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.098780 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.098786 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.098790 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.098794 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.098798 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.098802 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.098807 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.098811 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.098815 | controller |       09:28:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.098819 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.098824 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.098833 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.098838 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.098842 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.098846 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.098851 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.098855 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.098859 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.098863 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.098867 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.098871 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.098876 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.098882 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.098886 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.098893 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.098897 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.098901 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.098906 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.098910 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.098914 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.098918 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.098923 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.098927 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.098931 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.098935 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.098939 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.098944 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.098948 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.098952 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.098956 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.098960 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.098965 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.098969 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.098973 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.098977 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.098982 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.098986 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.098990 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.098994 | controller |       09:28:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.098999 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.099003 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.099007 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.099011 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.099023 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.099027 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.099033 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.099039 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.099045 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.099051 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.099057 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.099065 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.099072 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.099078 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.099083 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.099087 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.099091 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.099095 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.099100 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.099104 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.099108 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.099112 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.099117 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.099121 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.099125 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.099129 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.099133 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.099138 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.099142 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.099146 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.099150 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.099155 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.099159 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.099163 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.099167 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.099172 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.099176 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.099182 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.099186 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.099190 | controller |       09:28:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.099195 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.099199 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.099203 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.099207 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.099212 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.099223 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.099227 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.099232 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.099236 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.099240 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.099244 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.099249 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.099253 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.099257 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.099261 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.099265 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.099270 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.099276 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.099280 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.099285 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.099303 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.099310 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.099314 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.099318 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.099323 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.099327 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.099331 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.099335 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.099341 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.099347 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.099353 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.099357 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.099361 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.099365 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.099369 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.099374 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.099378 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.099382 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.099386 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.099390 | controller |       09:28:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.099395 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.099399 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.099403 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.099407 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.099411 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.099416 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.099426 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.099431 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.099435 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.099439 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.099444 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.099448 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.099452 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.099456 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.099460 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.099465 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.099469 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.099473 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.099477 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.099481 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.099488 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.099492 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.099497 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.099501 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.099505 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.099509 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.099513 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.099518 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.099522 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.099526 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.099530 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.099534 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.099539 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.099543 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.099547 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.099551 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.099556 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.099560 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.099564 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.099568 | controller |       09:28:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.099573 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.099577 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.099581 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.099585 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.099589 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.099593 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.099598 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.099602 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.099611 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.099616 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.099622 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.099626 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.099631 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.099636 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.099641 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.099645 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.099649 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.099653 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.099657 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.099662 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.099667 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.099671 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.099676 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.099680 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.099684 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.099688 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.099693 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.099697 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.099701 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.099705 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.099709 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.099714 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.099718 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.099722 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.099726 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.099731 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.099735 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.099739 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.099743 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.099749 | controller |       09:28:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.099753 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.099758 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.099762 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.099766 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.099773 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.099777 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.099782 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.099786 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.099790 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.099800 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.099804 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.099808 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.099813 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.099817 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.099821 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.099825 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.099830 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.099834 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.099838 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.099842 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.099846 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.099851 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.099855 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.099859 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.099863 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.099867 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.099872 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.099879 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.099883 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.099887 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.099892 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.099896 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.099900 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.099905 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.099909 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.099913 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.099917 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.099923 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.099927 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.099931 | controller |       09:28:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.099936 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.099940 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.099944 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.099948 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.099953 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.099957 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.099961 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.099965 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.099969 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.099974 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.099978 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.099987 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.099992 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.099996 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.100001 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.100005 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.100009 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.100013 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.100017 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.100022 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.100026 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.100030 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.100034 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.100038 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.100043 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.100047 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.100051 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.100055 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.100059 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.100065 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.100069 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.100074 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.100078 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.100082 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.100086 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.100091 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.100095 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.100099 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.100103 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.100107 | controller |       09:28:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.100112 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.100116 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.100120 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.100125 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.100131 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.100137 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.100143 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.100148 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.100152 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.100158 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.100162 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.100167 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.100178 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.100183 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.100187 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.100193 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.100197 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.100202 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.100206 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.100210 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.100216 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.100220 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.100225 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.100229 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.100233 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.100237 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.100241 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.100246 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.100250 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.100254 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.100258 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.100262 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.100266 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.100271 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.100275 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.100279 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.100283 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.100290 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.100310 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.100315 | controller |       09:28:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.100319 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.100323 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.100327 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.100332 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.100336 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.100340 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.100344 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.100349 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.100353 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.100357 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.100361 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.100366 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.100370 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.100376 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.100387 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.100391 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.100396 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.100400 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.100404 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.100408 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.100413 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.100417 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.100421 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.100425 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.100430 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.100438 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.100444 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.100450 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.100456 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.100462 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.100467 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.100471 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.100476 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.100480 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.100484 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.100488 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.100492 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.100497 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.100503 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.100509 | controller |       09:28:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.100515 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.100521 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.100525 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.100530 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.100536 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.100540 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.100544 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.100549 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.100553 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.100557 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.100561 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.100566 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.100570 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.100574 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.100578 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.100589 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.100595 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.100599 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.100603 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.100608 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.100614 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.100618 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.100622 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.100627 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.100631 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.100635 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.100639 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.100643 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.100647 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.100652 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.100657 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.100662 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.100666 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.100670 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.100674 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.100678 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.100683 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.100689 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.100693 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.100697 | controller |       09:28:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.100701 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.100706 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.100712 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.100718 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.100723 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.100732 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.100736 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.100741 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.100745 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.100749 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.100753 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.100758 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.100762 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.100766 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.100770 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.100774 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.100786 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.100791 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.100795 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.100800 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.100804 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.100808 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.100812 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.100816 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.100821 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.100825 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.100829 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.100833 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.100838 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.100844 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.100848 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.100852 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.100856 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.100861 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.100865 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.100869 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.100873 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.100878 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.100882 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.100886 | controller |       09:28:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.100890 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.100895 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.100899 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.100903 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.100907 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.100912 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.100916 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.100920 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.100924 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.100928 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.100933 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.100937 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.100941 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.100945 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.100949 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.100954 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.100958 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.100962 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.100972 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.100976 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.100984 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.100988 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.100992 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.100996 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.101002 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.101007 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.101012 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.101016 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.101020 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.101025 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.101029 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.101033 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.101037 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.101042 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.101046 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.101050 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.101054 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.101059 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.101063 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.101067 | controller |       09:28:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.101071 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.101075 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.101080 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.101084 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.101090 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.101094 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.101098 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.101102 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.101106 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.101111 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.101115 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.101119 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.101123 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.101129 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.101133 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.101137 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.101142 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.101147 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.101153 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.101166 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.101172 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.101177 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.101181 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.101185 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.101190 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.101196 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.101202 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.101208 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.101214 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.101220 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.101226 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.101232 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.101236 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.101240 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.101244 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.101249 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.101253 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.101257 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.101261 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.101266 | controller |       09:28:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.101270 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.101274 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.101278 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.101286 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.101308 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.101313 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.101317 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.101321 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.101325 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.101329 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.101334 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.101338 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.101342 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.101346 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.101350 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.101355 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.101359 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.101363 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.101367 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.101371 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.101384 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.101389 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.101394 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.101398 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.101402 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.101406 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.101410 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.101415 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.101419 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.101423 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.101427 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.101431 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.101436 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.101440 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.101444 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.101448 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.101453 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.101475 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.101480 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.101484 | controller |       09:28:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.101488 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.101492 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.101497 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.101501 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.101505 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.101509 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.101514 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.101518 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.101522 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.101526 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.101530 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.101535 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.101539 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.101543 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.101547 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.101553 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.101558 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.101562 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.101566 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.101570 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.101576 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.101588 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.101592 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.101598 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.101602 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.101606 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.101611 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.101615 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.101619 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.101625 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.101629 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.101633 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.101638 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.101642 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.101646 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.101650 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.101655 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.101659 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.101663 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.101667 | controller |       09:28:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.101672 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.101676 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.101680 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.101684 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.101688 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.101693 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.101697 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.101701 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.101705 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.101709 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.101714 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.101718 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.101722 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.101726 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.101730 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.101735 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.101739 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.101743 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.101747 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.101751 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.101757 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.101761 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.101766 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.101776 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.101781 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.101785 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.101789 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.101793 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.101798 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.101802 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.101806 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.101810 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.101814 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.101819 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.101823 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.101828 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.101833 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.101839 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.101845 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.101852 | controller |       09:28:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.101858 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.101864 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.101874 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.101881 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.101888 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.101895 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.101902 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.101909 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.101916 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.101922 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.101929 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.101936 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.101942 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.101950 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.101955 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.101959 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.101963 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.101967 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.101972 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.101976 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.101980 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.101984 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.101989 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.101993 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.102007 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.102012 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.102016 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.102020 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.102025 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.102029 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.102041 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.102045 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.102050 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.102054 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.102058 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.102062 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.102067 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.102071 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.102075 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.102079 | controller |       09:28:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.102084 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.102088 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.102092 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.102096 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.102104 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.102108 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.102112 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.102116 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.102121 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.102125 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.102129 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.102133 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.102138 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.102142 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.102146 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.102150 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.102154 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.102158 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.102163 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.102167 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.102171 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.102177 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.102181 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.102185 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.102189 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.102194 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.102205 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.102210 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.102214 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.102218 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.102222 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.102227 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.102231 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.102235 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.102239 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.102243 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.102248 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.102254 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.102258 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.102262 | controller |       09:28:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.102267 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.102271 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.102275 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.102279 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.102283 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.102290 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.102314 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.102319 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.102324 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.102328 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.102332 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.102336 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.102341 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.102345 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.102349 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.102353 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.102357 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.102361 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.102366 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.102370 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.102374 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.102378 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.102382 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.102387 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.102391 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.102395 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.102399 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.102411 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.102415 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.102422 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.102427 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.102431 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.102435 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.102439 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.102443 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.102448 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.102452 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.102456 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.102460 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.102464 | controller |       09:28:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.102469 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.102475 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.102479 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.102483 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.102489 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.102493 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.102497 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.102501 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.102506 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.102510 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.102514 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.102518 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.102522 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.102527 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.102531 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.102535 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.102539 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.102543 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.102548 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.102552 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.102559 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.102563 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.102567 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.102571 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.102576 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.102580 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.102584 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.102588 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.102592 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.102603 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.102608 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.102612 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.102616 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.102621 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.102625 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.102629 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.102633 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.102637 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.102641 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.102646 | controller |       09:28:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.102650 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.102654 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.102658 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.102662 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.102667 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.102671 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.102675 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.102679 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.102683 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.102687 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.102692 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.102696 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.102700 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.102706 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.102710 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.102714 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.102720 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.102724 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.102729 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.102733 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.102737 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.102741 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.102745 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.102750 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.102754 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.102758 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.102762 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.102766 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.102770 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.102775 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.102785 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.102789 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.102794 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.102798 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.102802 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.102807 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.102811 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.102815 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.102819 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.102823 | controller |       09:28:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.102828 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.102832 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.102836 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.102840 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.102846 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.102851 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.102855 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.102859 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.102863 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.102868 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.102872 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.102876 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.102880 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.102885 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.102889 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.102894 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.102899 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.102903 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.102908 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.102913 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.102917 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.102921 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.102926 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.102930 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.102934 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.102938 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.102943 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.102947 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.102951 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.102955 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.102960 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.102970 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.102975 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.102980 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.102986 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.102992 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.102998 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.103007 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.103016 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.103024 | controller |       09:28:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.103098 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.103104 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.103108 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.103113 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.103117 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.103121 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.103125 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.103129 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.103134 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.103138 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.103142 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.103146 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.103150 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.103155 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.103159 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.103163 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.103167 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.103171 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.103176 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.103180 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.103184 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.103188 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.103192 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.103197 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.103201 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.103205 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.103209 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.103213 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.103240 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.103250 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.103255 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.103259 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.103264 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.103277 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.103282 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.103286 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.103310 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.103315 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.103320 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.103324 | controller |       09:28:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.103328 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.103333 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.103337 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.103341 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.103346 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.103350 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.103354 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.103358 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.103363 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.103370 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.103375 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.103379 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.103383 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.103388 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.103397 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.103403 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.103409 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.103414 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.103418 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.103422 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.103428 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.103432 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.103436 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.103441 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.103445 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.103449 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.103453 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.103458 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.103462 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.103466 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.103472 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.103476 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.103480 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.103484 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.103497 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.103502 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.103506 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.103511 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.103515 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.103519 | controller |       09:28:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.103523 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.103528 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.103532 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.103536 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.103540 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.103544 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.103549 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.103553 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.103557 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.103561 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.103565 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.103569 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.103574 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.103580 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.103584 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.103588 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.103592 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.103596 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.103601 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.103605 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.103611 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.103615 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.103619 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.103623 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.103628 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.103632 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.103636 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.103667 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.103673 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.103677 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.103681 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.103686 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.103690 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.103694 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.103698 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.103710 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.103715 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.103719 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.103726 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.103730 | controller |       09:28:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.103734 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.103738 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.103743 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.103747 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.103754 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.103758 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.103762 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.103766 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.103770 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.103775 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.103779 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.103783 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.103787 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.103791 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.103795 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.103800 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.103804 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.103808 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.103812 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.103816 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.103821 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.103825 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.103829 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.103833 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.103837 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.103842 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.103846 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.103850 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.103854 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.103858 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.103862 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.103867 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.103871 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.103875 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.103879 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.103883 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.103888 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.103900 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.103906 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.103910 | controller |       09:28:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.103915 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.103919 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.103923 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.103927 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.103933 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.103938 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.103942 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.103946 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.103951 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.103955 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.103959 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.103964 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.103968 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.103974 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.103978 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.103982 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.103987 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.103992 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.103996 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.104000 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.104004 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.104009 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.104013 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.104017 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.104021 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.104026 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.104030 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.104034 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.104039 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.104044 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.104049 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.104053 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.104057 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.104062 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.104066 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.104070 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.104074 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.104079 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.104089 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.104094 | controller |       09:28:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.104099 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.104103 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.104107 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.104112 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.104116 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.104122 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.104127 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.104133 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.104140 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.104146 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.104152 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.104159 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.104166 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.104174 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.104181 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.104188 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.104195 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.104202 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.104209 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.104217 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.104227 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.104233 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.104237 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.104242 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.104246 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.104250 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.104254 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.104259 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.104263 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.104267 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.104271 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.104275 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.104279 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.104284 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.104309 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.104319 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.104324 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.104328 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.104332 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.104344 | controller |       09:28:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.104350 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.104354 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.104359 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.104363 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.104367 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.104371 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.104376 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.104380 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.104384 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.104388 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.104392 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.104397 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.104401 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.104408 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.104412 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.104418 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.104422 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.104426 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.104431 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.104435 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.104439 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.104443 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.104447 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.104452 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.104456 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.104460 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.104464 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.104468 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.104473 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.104477 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.104481 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.104485 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.104489 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.104494 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.104498 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.104502 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.104506 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.104510 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.104515 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.104519 | controller |       09:28:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.104530 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.104535 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.104540 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.104545 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.104553 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.104558 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.104564 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.104569 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.104574 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.104579 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.104585 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.104590 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.104597 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.104603 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.104608 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.104613 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.104619 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.104624 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.104629 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.104634 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.104642 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.104647 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.104652 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.104658 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.104663 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.104668 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.104674 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.104679 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.104685 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.104691 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.104696 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.104702 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.104707 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.104713 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.104719 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.104725 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.104731 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.104738 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.104745 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.104750 | controller |       09:28:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.104756 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.104762 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.104777 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.104784 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.104790 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.104795 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.104800 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.104804 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.104808 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.104812 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.104817 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.104821 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.104825 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.104829 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.104834 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.104838 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.104842 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.104846 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.104852 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.104857 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.104863 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.104867 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.104871 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.104879 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.104884 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.104890 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.104895 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.104899 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.104904 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.104910 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.104917 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.104922 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.104928 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.104932 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.104936 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.104940 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.104945 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.104949 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.104953 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.104958 | controller |       09:28:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.104963 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.104969 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.104974 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.104986 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.104991 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.104995 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.104999 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.105004 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.105008 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.105012 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.105017 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.105021 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.105025 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.105029 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.105034 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.105038 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.105042 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.105046 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.105051 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.105055 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.105062 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.105066 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.105071 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.105075 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.105079 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.105083 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.105088 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.105092 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.105096 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.105100 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.105105 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.105109 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.105113 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.105117 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.105122 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.105126 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.105130 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.105134 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.105139 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.105143 | controller |       09:28:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.105147 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.105151 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.105156 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.105160 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.105164 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.105175 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.105180 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.105184 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.105189 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.105193 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.105197 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.105203 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.105208 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.105213 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.105218 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.105222 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.105226 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.105231 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.105236 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.105240 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.105244 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.105249 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.105253 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.105257 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.105261 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.105266 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.105272 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.105277 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.105283 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.105307 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.105319 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.105326 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.105333 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.105340 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.105347 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.105354 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.105360 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.105367 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.105373 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.105377 | controller |       09:28:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.105382 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.105386 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.105390 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.105394 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.105404 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.105411 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.105425 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.105430 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.105435 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.105439 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.105443 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.105448 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.105452 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.105456 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.105460 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.105464 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.105558 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.105572 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.105578 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.105584 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.105590 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.105596 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.105601 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.105606 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.105612 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.105617 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.105621 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.105625 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.105629 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.105634 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.105641 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.105645 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.105650 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.105654 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.105658 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.105662 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.105667 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.105673 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.105677 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.105681 | controller |       09:28:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.105686 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.105690 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.105694 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.105698 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.105702 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.105706 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.105711 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.105741 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.105747 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.105751 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.105756 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.105760 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.105764 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.105768 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.105772 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.105776 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.105781 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.105785 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.105789 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.105793 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.105799 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.105804 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.105808 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.105812 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.105816 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.105821 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.105825 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.105829 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.105833 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.105839 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.105843 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.105847 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.105852 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.105856 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.105860 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.105864 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.105868 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.105873 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.105877 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.105881 | controller |       09:28:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.105885 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.105890 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.105896 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.105901 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.105905 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.105909 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.105914 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.105919 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.105924 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.105936 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.105944 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.105948 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.105953 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.105957 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.105962 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.105968 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.105972 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.105976 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.105981 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.105985 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.105991 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.105995 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.105999 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.106004 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.106008 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.106012 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.106017 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.106023 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.106027 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.106032 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.106036 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.106040 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.106044 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.106048 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.106053 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.106057 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.106061 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.106065 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.106069 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.106074 | controller |       09:28:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.106078 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.106082 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.106086 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.106090 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.106095 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.106099 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.106103 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.106107 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.106111 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.106116 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.106127 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.106131 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.106135 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.106142 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.106146 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.106150 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.106154 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.106159 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.106165 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.106170 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.106176 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.106181 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.106185 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.106190 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.106194 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.106198 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.106202 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.106209 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.106213 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.106217 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.106222 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.106226 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.106230 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.106234 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.106239 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.106243 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.106247 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.106251 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.106256 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.106260 | controller |       09:28:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.106264 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.106269 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.106273 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.106277 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.106283 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.106301 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.106310 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.106314 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.106318 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.106323 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.106327 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.106331 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.106343 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.106348 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.106352 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.106356 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.106361 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.106365 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.106369 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.106373 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.106377 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.106382 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.106386 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.106392 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.106397 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.106402 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.106407 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.106412 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.106418 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.106424 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.106433 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.106440 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.106446 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.106452 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.106458 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.106463 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.106470 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.106480 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.106487 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.106493 | controller |       09:28:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.106500 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.106507 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.106513 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.106520 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.106526 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.106532 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.106538 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.106544 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.106549 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.106555 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.106560 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.106564 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.106568 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.106582 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.106587 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.106591 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.106596 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.106600 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.106604 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.106608 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.106614 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.106620 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.106625 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.106631 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.106636 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.106642 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.106648 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.106654 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.106660 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.106668 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.106674 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.106680 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.106684 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.106688 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.106693 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.106697 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.106701 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.106705 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.106709 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.106714 | controller |       09:28:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.106718 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:28:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.106722 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.106726 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.106730 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.106734 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.106739 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.106743 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.106747 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.106751 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.106755 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.106759 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.106764 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.106768 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.106772 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.106776 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.106789 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.106793 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.106798 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.106802 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.106806 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.106816 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.106820 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.106824 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.106828 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.106833 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.106838 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.106842 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.106847 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.106851 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.106855 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.106859 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.106863 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.106868 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.106872 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.106876 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.106880 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.106884 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.106888 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:28:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.106893 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.106897 | controller |       09:28:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.106901 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.106905 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.106909 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.106914 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.106920 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.106924 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.106928 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.106932 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.106936 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.106940 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.106945 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.106949 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.106953 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.106959 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.106963 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.106967 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.106978 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.106983 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.106987 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.106991 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.106995 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.107000 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.107004 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.107008 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.107012 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.107016 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.107021 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.107025 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.107029 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.107033 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.107037 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.107041 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.107046 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.107050 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.107054 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.107058 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.107062 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.107067 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.107071 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.107075 | controller |       09:29:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.107079 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.107083 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.107088 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.107092 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.107098 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.107102 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.107106 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.107110 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.107116 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.107120 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.107124 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.107128 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.107133 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.107137 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.107141 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.107145 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.107149 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.107160 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.107165 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.107169 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.107173 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.107177 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.107182 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.107186 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.107190 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.107194 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.107198 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.107202 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.107207 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.107211 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.107215 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.107219 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.107223 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.107227 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.107232 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.107236 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.107240 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.107251 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.107255 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.107259 | controller |       09:29:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.107264 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.107268 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.107272 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.107276 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.107280 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.107285 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.107304 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.107312 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.107316 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.107320 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.107324 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.107329 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.107333 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.107337 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.107341 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.107347 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.107352 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.107356 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.107360 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.107371 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.107376 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.107380 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.107385 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.107389 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.107395 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.107399 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.107403 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.107407 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.107411 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.107418 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.107422 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.107426 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.107436 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.107441 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.107446 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.107451 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.107455 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.107459 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.107463 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.107468 | controller |       09:29:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.107472 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.107476 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.107480 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.107484 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.107489 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.107493 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.107497 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.107501 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.107506 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.107510 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.107514 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.107518 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.107522 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.107527 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.107531 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.107535 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.107539 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.107543 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.107548 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.107552 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.107564 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.107569 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.107574 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.107580 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.107585 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.107592 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.107598 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.107604 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.107610 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.107617 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.107624 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.107630 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.107637 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.107643 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.107650 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.107657 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.107664 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.107670 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.107676 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.107682 | controller |       09:29:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.107687 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.107693 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.107698 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.107707 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.107713 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.107719 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.107723 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.107729 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.107734 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.107740 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.107745 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.107750 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.107756 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.107764 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.107769 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.107775 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.107780 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.107786 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.107791 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.107797 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.107804 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.107820 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.107825 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.107830 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.107834 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.107838 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.107842 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.107847 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.107851 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.107855 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.107862 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.107867 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.107871 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.107875 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.107879 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.107884 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.107888 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.107892 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.107898 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.107902 | controller |       09:29:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.107906 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.107910 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.107915 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.107919 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.107925 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.107929 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.107933 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.107937 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.107942 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.107946 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.107950 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.107954 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.107958 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.107962 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.107967 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.107971 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.107975 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.107979 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.107983 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.107987 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.107992 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.107996 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.108000 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.108013 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.108018 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.108022 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.108026 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.108031 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.108035 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.108039 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.108043 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.108047 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.108051 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.108056 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.108060 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.108064 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.108068 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.108074 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.108078 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.108082 | controller |       09:29:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.108086 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.108091 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.108095 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.108099 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.108103 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.108107 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.108112 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.108116 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.108120 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.108124 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.108128 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.108133 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.108137 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.108141 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.108145 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.108149 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.108153 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.108158 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.108162 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.108166 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.108170 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.108174 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.108178 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.108183 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.108193 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.108201 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.108205 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.108210 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.108214 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.108220 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.108224 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.108228 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.108232 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.108236 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.108241 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.108245 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.108249 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.108253 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.108257 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.108262 | controller |       09:29:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.108266 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.108270 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.108276 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.108280 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.108285 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.108306 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.108311 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.108315 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.108319 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.108323 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.108328 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.108332 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.108336 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.108340 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.108344 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.108348 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.108353 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.108357 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.108361 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.108365 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.108371 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.108376 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.108380 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.108384 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.108388 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.108393 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.108451 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.108458 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.108463 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.108467 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.108471 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.108475 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.108480 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.108484 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.108488 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.108492 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.108497 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.108501 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.108505 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.108509 | controller |       09:29:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.108514 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.108518 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.108522 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.108526 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.108531 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.108535 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.108539 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.108543 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.108548 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.108553 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.108557 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.108561 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.108565 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.108572 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.108577 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.108581 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.108585 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.108590 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.108594 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.108598 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.108602 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.108609 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.108613 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.108618 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.108622 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.108627 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.108631 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.108643 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.108647 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.108652 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.108656 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.108660 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.108665 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.108669 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.108673 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.108678 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.108682 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.108686 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.108690 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.108695 | controller |       09:29:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.108699 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.108703 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.108708 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.108714 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.108722 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.108728 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.108735 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.108741 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.108748 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.108755 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.108762 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.108769 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.108775 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.108780 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.108786 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.108794 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.108800 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.108805 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.108811 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.108816 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.108824 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.108830 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.108835 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.108841 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.108847 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.108852 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.108858 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.108864 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.108870 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.108886 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.108892 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.108899 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.108904 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.108943 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.108950 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.108956 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.108962 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.108972 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.108979 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.108985 | controller |       09:29:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.108992 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.109001 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.109008 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.109015 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.109021 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.109028 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.109035 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.109042 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.109049 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.109056 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.109063 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.109071 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.109078 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.109085 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.109091 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.109096 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.109100 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.109105 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.109109 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.109113 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.109118 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.109122 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.109126 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.109131 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.109135 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.109139 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.109143 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.109147 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.109152 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.109159 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.109173 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.109178 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.109183 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.109187 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.109192 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.109196 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.109200 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.109205 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.109209 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.109213 | controller |       09:29:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.109217 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.109222 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.109226 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.109230 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.109234 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.109239 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.109243 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.109247 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.109251 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.109256 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.109261 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.109265 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.109269 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.109274 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.109278 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.109282 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.109304 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.109312 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.109316 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.109320 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.109326 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.109331 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.109335 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.109339 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.109343 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.109348 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.109352 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.109356 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.109360 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.109364 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.109370 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.109382 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.109387 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.109391 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.109396 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.109400 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.109404 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.109408 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.109413 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.109417 | controller |       09:29:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.109421 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.109425 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.109430 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.109434 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.109438 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.109442 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.109446 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.109451 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.109455 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.109459 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.109463 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.109467 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.109472 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.109500 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.109507 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.109512 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.109518 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.109524 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.109530 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.109535 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.109540 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.109546 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.109551 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.109557 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.109562 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.109566 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.109570 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.109574 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.109578 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.109582 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.109587 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.109591 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.109595 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.109607 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.109611 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.109616 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.109620 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.109624 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.109628 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.109633 | controller |       09:29:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.109639 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.109644 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.109648 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.109652 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.109658 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.109662 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.109667 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.109671 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.109675 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.109679 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.109683 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.109687 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.109692 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.109696 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.109700 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.109704 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.109708 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.109713 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.109717 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.109721 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.109725 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.109729 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.109733 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.109738 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.109742 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.109746 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.109750 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.109754 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.109759 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.109763 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.109767 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.109771 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.109775 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.109780 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.109790 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.109795 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.109799 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.109805 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.109809 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.109813 | controller |       09:29:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.109818 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.109822 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.109826 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.109830 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.109836 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.109840 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.109844 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.109849 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.109853 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.109857 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.109861 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.109866 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.109870 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.109874 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.109880 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.109884 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.109888 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.109893 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.109897 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.109901 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.109907 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.109911 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.109915 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.109919 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.109923 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.109927 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.109932 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.109936 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.109940 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.109947 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.109951 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.109955 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.109959 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.109964 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.109968 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.109979 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.109983 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.109988 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.109992 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.109996 | controller |       09:29:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.110001 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.110005 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.110009 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.110013 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.110017 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.110022 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.110026 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.110030 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.110034 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.110038 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.110043 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.110047 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.110051 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.110055 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.110059 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.110063 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.110068 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.110072 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.110076 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.110080 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.110088 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.110092 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.110096 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.110100 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.110105 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.110109 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.110113 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.110117 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.110121 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.110127 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.110132 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.110138 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.110144 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.110151 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.110157 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.110163 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.110187 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.110196 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.110203 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.110210 | controller |       09:29:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.110217 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.110223 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.110227 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.110231 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.110235 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.110242 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.110246 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.110252 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.110256 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.110260 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.110264 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.110269 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.110273 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.110279 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.110283 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.110303 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.110312 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.110316 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.110320 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.110325 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.110329 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.110333 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.110337 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.110341 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.110346 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.110350 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.110354 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.110358 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.110362 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.110367 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.110371 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.110375 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.110379 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.110384 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.110388 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.110392 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.110396 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.110400 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.110414 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.110418 | controller |       09:29:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.110422 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.110427 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.110431 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.110435 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.110442 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.110446 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.110450 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.110454 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.110458 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.110463 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.110467 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.110471 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.110475 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.110481 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.110486 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.110490 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.110494 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.110498 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.110503 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.110507 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.110511 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.110515 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.110519 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.110523 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.110528 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.110532 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.110536 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.110540 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.110544 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.110549 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.110553 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.110557 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.110561 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.110565 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.110569 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.110574 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.110578 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.110584 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.110588 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.110599 | controller |       09:29:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.110604 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.110608 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.110613 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.110617 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.110621 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.110625 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.110629 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.110634 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.110638 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.110642 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.110646 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.110650 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.110655 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.110659 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.110663 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.110667 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.110671 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.110675 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.110680 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.110684 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.110688 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.110692 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.110696 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.110701 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.110705 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.110709 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.110713 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.110717 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.110721 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.110728 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.110733 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.110737 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.110742 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.110746 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.110750 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.110756 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.110760 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.110764 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.110768 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.110772 | controller |       09:29:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.110783 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.110788 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.110792 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.110797 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.110801 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.110805 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.110809 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.110814 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.110818 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.110822 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.110826 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.110830 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.110834 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.110839 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.110843 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.110847 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.110851 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.110855 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.110860 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.110864 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.110871 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.110875 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.110879 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.110884 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.110888 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.110892 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.110896 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.110900 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.110904 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.110909 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.110913 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.110917 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.110921 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.110926 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.110931 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.110936 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.110940 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.110945 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.110950 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.110955 | controller |       09:29:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.110960 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.110964 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.110975 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.110979 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.110984 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.110988 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.110992 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.110996 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.111001 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.111005 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.111009 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.111013 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.111019 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.111025 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.111029 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.111034 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.111038 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.111042 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.111046 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.111051 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.111055 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.111059 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.111063 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.111068 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.111072 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.111076 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.111080 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.111085 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.111089 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.111093 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.111097 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.111101 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.111106 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.111110 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.111114 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.111119 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.111123 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.111127 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.111131 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.111136 | controller |       09:29:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.111140 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.111144 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.111148 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.111160 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.111169 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.111174 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.111178 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.111183 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.111187 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.111191 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.111196 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.111200 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.111204 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.111208 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.111213 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.111217 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.111221 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.111225 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.111230 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.111234 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.111238 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.111243 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.111247 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.111251 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.111256 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.111261 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.111267 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.111274 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.111280 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.111289 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.111317 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.111325 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.111337 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.111345 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.111351 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.111356 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.111361 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.111367 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.111371 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.111377 | controller |       09:29:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.111381 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.111385 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.111390 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.111394 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.111398 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.111412 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.111417 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.111422 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.111426 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.111430 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.111435 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.111439 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.111443 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.111447 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.111452 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.111456 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.111460 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.111464 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.111469 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.111473 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.111477 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.111481 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.111485 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.111490 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.111494 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.111498 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.111502 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.111506 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.111511 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.111516 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.111521 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.111525 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.111529 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.111533 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.111538 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.111542 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.111546 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.111550 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.111554 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.111558 | controller |       09:29:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.111563 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.111567 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.111571 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.111575 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.111579 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.111584 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.111594 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.111599 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.111603 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.111607 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.111612 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.111634 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.111639 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.111643 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.111647 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.111653 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.111657 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.111661 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.111666 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.111670 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.111677 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.111681 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.111685 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.111689 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.111694 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.111698 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.111702 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.111706 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.111710 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.111714 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.111718 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.111723 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.111727 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.111731 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.111735 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.111740 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.111744 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.111748 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.111752 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.111756 | controller |       09:29:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.111761 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.111765 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.111769 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.111773 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.111777 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.111782 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.111786 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.111797 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.111802 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.111806 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.111810 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.111815 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.111819 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.111825 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.111829 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.111833 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.111838 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.111842 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.111846 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.111850 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.111856 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.111861 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.111865 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.111869 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.111873 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.111877 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.111881 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.111886 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.111891 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.111895 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.111900 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.111904 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.111909 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.111914 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.111919 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.111923 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.111928 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.111934 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.111938 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.111942 | controller |       09:29:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.111947 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.111951 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.111955 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.111959 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.111965 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.111969 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.111973 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.111978 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.111982 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.111993 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.111997 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.112002 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.112006 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.112010 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.112014 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.112019 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.112023 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.112027 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.112031 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.112036 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.112040 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.112044 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.112048 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.112052 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.112057 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.112061 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.112065 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.112069 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.112073 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.112077 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.112083 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.112087 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.112092 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.112096 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.112100 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.112104 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.112108 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.112114 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.112136 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.112140 | controller |       09:29:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.112145 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.112149 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.112153 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.112157 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.112161 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.112166 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.112170 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.112174 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.112178 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.112182 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.112195 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.112201 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.112205 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.112209 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.112213 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.112218 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.112222 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.112226 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.112230 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.112235 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.112239 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.112243 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.112247 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.112251 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.112255 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.112260 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.112264 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.112268 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.112272 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.112279 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.112284 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.112309 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.112316 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.112320 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.112325 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.112329 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.112333 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.112337 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.112342 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.112346 | controller |       09:29:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.112350 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.112354 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.112359 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.112363 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.112367 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.112371 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.112376 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.112380 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.112384 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.112390 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.112396 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.112402 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.112417 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.112425 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.112432 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.112440 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.112447 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.112454 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.112459 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.112465 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.112476 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.112483 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.112490 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.112497 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.112504 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.112510 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.112515 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.112523 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.112527 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.112531 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.112535 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.112539 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.112544 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.112548 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.112552 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.112556 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.112560 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.112565 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.112570 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.112575 | controller |       09:29:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.112579 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.112583 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.112587 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.112591 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.112597 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.112601 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.112605 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.112609 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.112614 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.112618 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.112622 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.112626 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.112630 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.112646 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.112652 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.112656 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.112660 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.112664 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.112669 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.112673 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.112677 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.112681 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.112685 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.112689 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.112694 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.112698 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.112702 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.112706 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.112710 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.112714 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.112719 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.112723 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.112727 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.112731 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.112735 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.112740 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.112744 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.112748 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.112752 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.112756 | controller |       09:29:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.112761 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.112765 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.112769 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.112773 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.112779 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.112783 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.112787 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.112792 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.112796 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.112802 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.112807 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.112811 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.112815 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.112819 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.112823 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.112834 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.112838 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.112843 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.112847 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.112851 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.112857 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.112861 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.112866 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.112870 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.112874 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.112878 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.112882 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.112887 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.112891 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.112895 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.112899 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.112904 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.112908 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.112912 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.112916 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.112921 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.112925 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.112931 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.112935 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.112939 | controller |       09:29:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.112944 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.112948 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.112953 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.112957 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.112961 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.112965 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.112969 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.112974 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.112978 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.112982 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.112986 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.112990 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.112995 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.112999 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.113003 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.113009 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.113019 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.113024 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.113028 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.113032 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.113037 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.113041 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.113045 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.113049 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.113054 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.113059 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.113064 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.113068 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.113072 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.113078 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.113082 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.113086 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.113090 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.113094 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.113099 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.113103 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.113107 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.113111 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.113115 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.113120 | controller |       09:29:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.113124 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.113128 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.113132 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.113136 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.113141 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.113145 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.113149 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.113153 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.113157 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.113161 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.113166 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.113170 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.113174 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.113178 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.113182 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.113187 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.113191 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.113201 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.113206 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.113210 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.113217 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.113221 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.113226 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.113230 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.113234 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.113239 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.113243 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.113247 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.113251 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.113256 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.113260 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.113264 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.113269 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.113324 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.113335 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.113340 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.113344 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.113349 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.113353 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.113357 | controller |       09:29:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.113361 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.113366 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.113370 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.113374 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.113378 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.113383 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.113387 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.113391 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.113398 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.113403 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.113407 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.113411 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.113416 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.113424 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.113428 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.113433 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.113437 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.113441 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.113454 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.113458 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.113463 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.113467 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.113472 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.113476 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.113480 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.113485 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.113521 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.113527 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.113532 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.113538 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.113546 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.113551 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.113557 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.113563 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.113568 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.113574 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.113579 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.113585 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.113590 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.113597 | controller |       09:29:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.113603 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.113612 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.113619 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.113626 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.113635 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.113643 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.113651 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.113659 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.113666 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.113673 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.113681 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.113688 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.113694 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.113699 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.113703 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.113707 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.113713 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.113717 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.113721 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.113725 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.113740 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.113745 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.113780 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.113785 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.113811 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.113816 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.113820 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.113824 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.113829 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.113833 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.113837 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.113841 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.113845 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.113850 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.113854 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.113858 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.113862 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.113869 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.113873 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.113877 | controller |       09:29:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.113882 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.113886 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.113890 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.113894 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.113899 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.113903 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.113907 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.113911 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.113915 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.113919 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.113924 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.113928 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.113932 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.113936 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.113940 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.113944 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.113948 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.113953 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.113957 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.113961 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.113967 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.113979 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.113983 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.113987 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.113992 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.113996 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.114001 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.114005 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.114009 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.114015 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.114019 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.114023 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.114028 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.114032 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.114036 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.114040 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.114044 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.114049 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.114053 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.114057 | controller |       09:29:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.114062 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.114066 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.114070 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.114075 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.114081 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.114085 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.114089 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.114093 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.114098 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.114102 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.114106 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.114110 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.114114 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.114119 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.114123 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.114127 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.114131 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.114135 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.114140 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.114144 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.114149 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.114154 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.114164 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.114169 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.114173 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.114178 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.114182 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.114186 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.114190 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.114195 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.114199 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.114203 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.114207 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.114212 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.114216 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.114220 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.114224 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.114228 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.114233 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.114237 | controller |       09:29:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.114241 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.114246 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.114250 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.114254 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.114258 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.114262 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.114267 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.114271 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.114275 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.114279 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.114284 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.114304 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.114311 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.114317 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.114322 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.114326 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.114330 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.114334 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.114338 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.114343 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.114347 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.114351 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.114355 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.114361 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.114373 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.114378 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.114382 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.114386 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.114391 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.114395 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.114399 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.114403 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.114407 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.114411 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.114416 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.114420 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.114424 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.114428 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.114432 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.114437 | controller |       09:29:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.114441 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.114445 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.114449 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.114453 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.114459 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.114464 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.114468 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.114472 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.114476 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.114480 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.114485 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.114489 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.114493 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.114497 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.114501 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.114507 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.114511 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.114515 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.114520 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.114524 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.114528 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.114532 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.114536 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.114541 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.114545 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.114555 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.114560 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.114564 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.114569 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.114573 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.114577 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.114581 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.114586 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.114590 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.114594 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.114598 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.114602 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.114609 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.114614 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.114618 | controller |       09:29:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.114622 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.114626 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.114632 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.114636 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.114640 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.114645 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.114649 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.114653 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.114657 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.114661 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.114666 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.114670 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.114674 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.114678 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.114682 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.114686 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.114691 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.114695 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.114699 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.114703 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.114709 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.114713 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.114717 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.114721 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.114726 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.114731 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.114737 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.114752 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.114760 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.114771 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.114779 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.114786 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.114794 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.114801 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.114809 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.114817 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.114823 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.114830 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.114836 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.114842 | controller |       09:29:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.114848 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.114853 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.114859 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.114864 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.114870 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.114879 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.114883 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.114888 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.114892 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.114896 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.114900 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.114905 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.114909 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.114913 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.114917 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.114922 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.114926 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.114930 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.114934 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.114939 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.114945 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.114951 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.114955 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.114960 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.114964 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.114968 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.114972 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.114978 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.114992 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.114997 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.115002 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.115006 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.115012 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.115016 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.115021 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.115025 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.115029 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.115033 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.115038 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.115042 | controller |       09:29:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.115046 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.115050 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.115055 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.115059 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.115063 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.115067 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.115072 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.115076 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.115080 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.115084 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.115088 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.115093 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.115097 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.115103 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.115107 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.115111 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.115116 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.115120 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.115124 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.115128 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.115133 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.115137 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.115141 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.115145 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.115150 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.115154 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.115158 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.115162 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.115167 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.115171 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.115182 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.115187 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.115191 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.115196 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.115200 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.115204 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.115209 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.115213 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.115217 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.115222 | controller |       09:29:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.115226 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.115232 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.115237 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.115241 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.115248 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.115253 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.115257 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.115261 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.115265 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.115269 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.115274 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.115278 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.115283 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.115303 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.115311 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.115315 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.115320 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.115325 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.115329 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.115333 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.115338 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.115342 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.115346 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.115350 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.115355 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.115359 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.115363 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.115367 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.115372 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.115376 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.115380 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.115394 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.115400 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.115404 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.115409 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.115413 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.115417 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.115424 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.115428 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.115432 | controller |       09:29:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.115436 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.115441 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.115445 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.115449 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.115492 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.115496 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.115502 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.115506 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.115510 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.115514 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.115519 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.115523 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.115527 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.115531 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.115535 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.115540 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.115546 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.115550 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.115554 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.115558 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.115562 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.115567 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.115571 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.115575 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.115594 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.115599 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.115603 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.115607 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.115612 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.115619 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.115623 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.115627 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.115655 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.115661 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.115665 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.115669 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.115673 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.115677 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.115681 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.115686 | controller |       09:29:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.115690 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.115694 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.115698 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.115702 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.115707 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.115722 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.115726 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.115730 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.115734 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.115738 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.115743 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.115747 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.115751 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.115755 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.115759 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.115763 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.115768 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.115772 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.115776 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.115780 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.115787 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.115791 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.115796 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.115800 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.115804 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.115808 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.115812 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.115817 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.115821 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.115825 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.115829 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.115833 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.115837 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.115855 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.115860 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.115864 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.115870 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.115876 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.115882 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.115888 | controller |       09:29:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.115897 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.115904 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.115911 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.115917 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.115924 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.115931 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.115938 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.115945 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.115951 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.115958 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.115964 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.115970 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.115975 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.115982 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.115986 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.115992 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.115997 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.116001 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.116005 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.116009 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.116015 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.116020 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.116024 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.116028 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.116032 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.116036 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.116040 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.116045 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.116049 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.116053 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.116057 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.116061 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.116066 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.116070 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.116074 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.116089 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.116094 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.116099 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.116103 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.116107 | controller |       09:29:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.116111 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.116115 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.116120 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.116124 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.116130 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.116134 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.116138 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.116143 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.116147 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.116151 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.116155 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.116159 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.116164 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.116168 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.116173 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.116178 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.116182 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.116186 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.116190 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.116194 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.116199 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.116203 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.116207 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.116211 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.116215 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.116219 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.116224 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.116228 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.116232 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.116236 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.116240 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.116245 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.116249 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.116253 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.116257 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.116261 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.116273 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.116279 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.116283 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.116301 | controller |       09:29:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.116310 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.116314 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.116318 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.116322 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.116326 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.116331 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.116335 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.116339 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.116343 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.116347 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.116352 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.116356 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.116360 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.116364 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.116368 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.116372 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.116377 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.116381 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.116385 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.116389 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.116393 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.116397 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.116402 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.116406 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.116410 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.116414 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.116418 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.116423 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.116427 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.116433 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.116439 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.116443 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.116447 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.116451 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.116456 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.116460 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.116464 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.116468 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.116483 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.116490 | controller |       09:29:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.116497 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.116503 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.116507 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.116512 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.116516 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.116520 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.116524 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.116528 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.116533 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.116537 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.116541 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.116545 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.116549 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.116554 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.116558 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.116562 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.116566 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.116570 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.116575 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.116579 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.116585 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.116589 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.116594 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.116598 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.116602 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.116606 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.116611 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.116616 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.116620 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.116624 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.116628 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.116632 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.116637 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.116641 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.116645 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.116649 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.116653 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.116657 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.116662 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.116673 | controller |       09:29:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.116678 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.116682 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.116686 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.116691 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.116695 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.116699 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.116703 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.116708 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.116712 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.116716 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.116720 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.116725 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.116729 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.116737 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.116741 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.116745 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.116750 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.116754 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.116758 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.116762 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.116767 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.116771 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.116775 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.116779 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.116783 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.116788 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.116792 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.116796 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.116800 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.116804 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.116809 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.116813 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.116817 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.116821 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.116825 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.116830 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.116834 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.116838 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.116842 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.116846 | controller |       09:29:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.116857 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.116862 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.116866 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.116870 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.116880 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.116884 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.116889 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.116893 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.116897 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.116901 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.116906 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.116910 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.116914 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.116918 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.116922 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.116927 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.116931 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.116935 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.116939 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.116944 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.116949 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.116954 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.116958 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.116962 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.116966 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.116970 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.116975 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.116979 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.116983 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.116987 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.116992 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.116996 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.117000 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.117005 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.117011 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.117019 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.117025 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.117033 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.117040 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.117047 | controller |       09:29:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.117054 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.117060 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.117077 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.117086 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.117093 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.117099 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.117104 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.117108 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.117113 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.117117 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.117121 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.117125 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.117129 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.117133 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.117138 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.117142 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.117146 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.117150 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.117154 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.117159 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.117163 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.117167 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.117171 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.117176 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.117180 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.117184 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.117188 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.117192 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.117197 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.117203 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.117207 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.117212 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.117216 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.117220 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.117224 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.117228 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.117233 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.117237 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.117243 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.117248 | controller |       09:29:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.117252 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:29:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.117256 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.117260 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.117273 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.117278 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.117282 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.117289 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.117312 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.117317 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.117321 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.117325 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.117329 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.117336 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.117341 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.117345 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.117350 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.117355 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.117359 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.117363 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.117367 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.117373 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.117378 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.117382 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.117386 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.117390 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.117394 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.117398 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.117403 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.117407 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.117411 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.117415 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.117419 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.117423 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.117428 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.117432 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.117436 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.117440 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.117444 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:29:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.117449 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.117455 | controller |       09:29:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.117470 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.117479 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.117485 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.117490 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.117534 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.117541 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.117547 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.117551 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.117555 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.117559 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.117564 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.117568 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.117572 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.117579 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.117583 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.117588 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.117592 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.117596 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.117600 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.117604 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.117609 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.117613 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.117617 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.117621 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.117625 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.117630 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.117634 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.117638 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.117642 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.117646 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.117650 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.117655 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.117661 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.117666 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.117670 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.117674 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.117678 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.117682 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.117687 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.117691 | controller |       09:30:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.117695 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.117699 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.117704 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.117708 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.117715 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.117719 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.117730 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.117735 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.117739 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.117743 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.117748 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.117752 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.117756 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.117760 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.117764 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.117768 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.117773 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.117777 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.117781 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.117785 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.117789 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.117793 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.117798 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.117802 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.117806 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.117810 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.117814 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.117818 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.117822 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.117827 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.117832 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.117837 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.117841 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.117845 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.117849 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.117853 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.117857 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.117863 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.117868 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.117872 | controller |       09:30:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.117876 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.117880 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.117884 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.117888 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.117893 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.117897 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.117901 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.117911 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.117916 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.117920 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.117924 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.117932 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.117936 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.117940 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.117944 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.117948 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.117952 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.117957 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.117961 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.117965 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.117970 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.117975 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.117979 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.117983 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.117987 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.117991 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.117995 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.118000 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.118004 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.118009 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.118014 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.118018 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.118022 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.118026 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.118030 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.118034 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.118039 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.118043 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.118047 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.118051 | controller |       09:30:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.118055 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.118059 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.118063 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.118068 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.118072 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.118076 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.118080 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.118084 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.118088 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.118099 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.118104 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.118108 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.118112 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.118117 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.118121 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.118125 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.118129 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.118133 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.118139 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.118147 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.118166 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.118174 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.118181 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.118188 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.118195 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.118201 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.118208 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.118215 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.118225 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.118232 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.118238 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.118243 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.118248 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.118252 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.118256 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.118260 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.118264 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.118269 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.118273 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.118277 | controller |       09:30:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.118281 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.118286 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.118308 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.118312 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.118319 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.118324 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.118328 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.118332 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.118336 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.118340 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.118354 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.118359 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.118363 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.118371 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.118375 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.118379 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.118383 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.118387 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.118391 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.118395 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.118400 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.118404 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.118408 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.118412 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.118416 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.118420 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.118425 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.118429 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.118433 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.118437 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.118441 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.118445 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.118451 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.118461 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.118475 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.118482 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.118487 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.118493 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.118498 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.118504 | controller |       09:30:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.118509 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.118515 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.118520 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.118526 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.118533 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.118537 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.118542 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.118546 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.118550 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.118554 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.118561 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.118565 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.118580 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.118586 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.118591 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.118595 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.118599 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.118603 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.118608 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.118612 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.118616 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.118620 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.118624 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.118629 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.118633 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.118637 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.118641 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.118645 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.118649 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.118654 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.118658 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.118662 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.118666 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.118670 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.118675 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.118679 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.118683 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.118689 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.118693 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.118697 | controller |       09:30:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.118701 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.118705 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.118710 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.118714 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.118718 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.118722 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.118726 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.118730 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.118735 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.118739 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.118743 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.118747 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.118751 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.118762 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.118767 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.118773 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.118777 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.118781 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.118786 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.118790 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.118794 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.118798 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.118802 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.118807 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.118811 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.118815 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.118819 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.118825 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.118829 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.118835 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.118839 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.118843 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.118847 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.118851 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.118856 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.118860 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.118864 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.118868 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.118872 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.118877 | controller |       09:30:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.118881 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.118885 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.118889 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.118893 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.118897 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.118902 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.118906 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.118910 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.118914 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.118919 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.118923 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.118927 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.118931 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.118935 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.118946 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.118950 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.118955 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.118960 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.118964 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.118968 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.118976 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.118980 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.118984 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.118988 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.118992 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.118996 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.119001 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.119005 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.119009 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.119013 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.119017 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.119021 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.119026 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.119030 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.119034 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.119038 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.119042 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.119046 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.119051 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.119055 | controller |       09:30:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.119059 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.119063 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.119067 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.119071 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.119075 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.119080 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.119084 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.119088 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.119092 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.119099 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.119104 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.119108 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.119112 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.119117 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.119122 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.119126 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.119137 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.119142 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.119146 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.119150 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.119155 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.119159 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.119163 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.119167 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.119171 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.119175 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.119180 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.119184 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.119188 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.119192 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.119198 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.119202 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.119206 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.119210 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.119215 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.119219 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.119223 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.119227 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.119231 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.119235 | controller |       09:30:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.119240 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.119244 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.119248 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.119252 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.119258 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.119262 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.119266 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.119270 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.119275 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.119281 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.119313 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.119331 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.119338 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.119345 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.119352 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.119359 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.119365 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.119383 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.119388 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.119392 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.119397 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.119401 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.119405 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.119409 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.119414 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.119422 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.119426 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.119430 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.119434 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.119439 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.119443 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.119448 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.119453 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.119469 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.119478 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.119483 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.119487 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.119494 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.119499 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.119503 | controller |       09:30:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.119507 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.119511 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.119516 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.119520 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.119524 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.119531 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.119535 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.119539 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.119544 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.119548 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.119552 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.119556 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.119560 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.119565 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.119569 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.119573 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.119577 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.119581 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.119596 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.119601 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.119605 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.119610 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.119614 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.119618 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.119622 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.119626 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.119631 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.119635 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.119639 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.119647 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.119651 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.119655 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.119659 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.119664 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.119668 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.119672 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.119676 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.119680 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.119684 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.119689 | controller |       09:30:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.119693 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.119697 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.119701 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.119705 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.119711 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.119715 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.119719 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.119723 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.119729 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.119733 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.119737 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.119741 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.119745 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.119750 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.119754 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.119758 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.119762 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.119766 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.119770 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.119781 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.119787 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.119791 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.119796 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.119800 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.119804 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.119808 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.119812 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.119816 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.119820 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.119825 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.119829 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.119833 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.119837 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.119841 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.119845 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.119850 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.119854 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.119858 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.119862 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.119866 | controller |       09:30:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.119870 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.119875 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.119879 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.119883 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.119887 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.119892 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.119896 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.119900 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.119904 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.119909 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.119913 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.119917 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.119922 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.119929 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.119933 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.119938 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.119943 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.119948 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.119953 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.119957 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.119962 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.119973 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.119977 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.119982 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.119987 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.119991 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.119996 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.120000 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.120004 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.120008 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.120012 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.120016 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.120021 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.120025 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.120029 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.120033 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.120037 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.120041 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.120045 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.120049 | controller |       09:30:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.120054 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.120058 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.120062 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.120066 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.120073 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.120078 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.120082 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.120086 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.120090 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.120094 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.120098 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.120102 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.120107 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.120111 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.120115 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.120137 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.120141 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.120145 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.120150 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.120154 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.120158 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.120162 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.120173 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.120178 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.120182 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.120186 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.120190 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.120194 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.120199 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.120203 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.120207 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.120211 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.120215 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.120219 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.120223 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.120227 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.120232 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.120237 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.120242 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.120246 | controller |       09:30:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.120250 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.120254 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.120258 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.120264 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.120268 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.120272 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.120276 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.120280 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.120285 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.120307 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.120314 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.120319 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.120323 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.120327 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.120331 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.120335 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.120339 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.120344 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.120348 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.120352 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.120356 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.120360 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.120364 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.120369 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.120380 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.120384 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.120389 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.120393 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.120397 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.120404 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.120408 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.120414 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.120419 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.120429 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.120443 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.120450 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.120456 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.120463 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.120469 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.120476 | controller |       09:30:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.120483 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.120490 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.120497 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.120504 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.120511 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.120518 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.120524 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.120530 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.120536 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.120540 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.120544 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.120548 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.120552 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.120557 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.120561 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.120568 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.120572 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.120577 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.120581 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.120585 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.120592 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.120596 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.120600 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.120606 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.120611 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.120626 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.120632 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.120636 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.120640 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.120644 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.120650 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.120654 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.120659 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.120663 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.120667 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.120671 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.120675 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.120679 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.120683 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.120688 | controller |       09:30:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.120692 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.120696 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.120700 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.120704 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.120708 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.120712 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.120717 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.120721 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.120725 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.120729 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.120733 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.120737 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.120741 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.120747 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.120751 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.120756 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.120760 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.120764 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.120768 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.120772 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.120776 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.120780 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.120785 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.120789 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.120793 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.120797 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.120801 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.120812 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.120817 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.120822 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.120826 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.120830 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.120834 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.120838 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.120843 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.120847 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.120851 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.120855 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.120859 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.120863 | controller |       09:30:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.120868 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.120872 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.120884 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.120888 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.120894 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.120898 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.120902 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.120906 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.120910 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.120915 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.120919 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.120923 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.120927 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.120931 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.120936 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.120940 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.120944 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.120948 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.120952 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.120957 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.120962 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.120967 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.120971 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.120975 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.120979 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.120983 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.120988 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.120992 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.121003 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.121007 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.121012 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.121016 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.121020 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.121025 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.121029 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.121033 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.121037 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.121044 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.121049 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.121053 | controller |       09:30:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.121057 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.121062 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.121067 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.121071 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.121077 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.121081 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.121085 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.121089 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.121094 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.121098 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.121102 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.121107 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.121111 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.121115 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.121120 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.121124 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.121128 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.121132 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.121137 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.121141 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.121145 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.121153 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.121157 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.121161 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.121166 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.121170 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.121174 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.121178 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.121183 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.121196 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.121200 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.121205 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.121209 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.121213 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.121218 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.121222 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.121226 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.121231 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.121235 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.121240 | controller |       09:30:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.121244 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.121248 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.121252 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.121257 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.121261 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.121265 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.121269 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.121274 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.121278 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.121282 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.121289 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.121310 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.121315 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.121319 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.121324 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.121328 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.121332 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.121336 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.121341 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.121345 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.121351 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.121356 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.121360 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.121364 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.121368 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.121372 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.121377 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.121381 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.121385 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.121389 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.121394 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.121406 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.121411 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.121415 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.121419 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.121424 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.121428 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.121432 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.121436 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.121440 | controller |       09:30:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.121445 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.121453 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.121459 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.121465 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.121471 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.121477 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.121483 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.121489 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.121493 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.121497 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.121502 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.121522 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.121527 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.121534 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.121538 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.121544 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.121548 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.121552 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.121557 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.121561 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.121565 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.121570 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.121575 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.121581 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.121587 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.121593 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.121599 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.121606 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.121612 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.121619 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.121626 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.121633 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.121651 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.121660 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.121667 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.121672 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.121678 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.121682 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.121686 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.121690 | controller |       09:30:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.121694 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.121699 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.121703 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.121707 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.121713 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.121718 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.121722 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.121726 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.121730 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.121735 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.121739 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.121743 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.121747 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.121751 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.121756 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.121760 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.121766 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.121770 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.121775 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.121779 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.121783 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.121787 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.121791 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.121796 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.121800 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.121804 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.121808 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.121812 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.121816 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.121821 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.121825 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.121829 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.121833 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.121845 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.121850 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.121854 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.121858 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.121875 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.121881 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.121885 | controller |       09:30:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.121889 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.121894 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.121898 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.121902 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.121906 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.121911 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.121916 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.121921 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.121926 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.121930 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.121935 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.121941 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.121945 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.121949 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.121953 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.121957 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.121961 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.121966 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.121970 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.121974 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.121980 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.121984 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.121988 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.121992 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.121996 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.122000 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.122005 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.122009 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.122013 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.122019 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.122024 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.122028 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.122033 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.122037 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.122041 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.122052 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.122057 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.122061 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.122065 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.122070 | controller |       09:30:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.122075 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.122080 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.122084 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.122088 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.122092 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.122096 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.122100 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.122105 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.122109 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.122113 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.122117 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.122121 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.122125 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.122129 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.122134 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.122138 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.122142 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.122146 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.122150 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.122154 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.122160 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.122164 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.122168 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.122173 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.122177 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.122181 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.122185 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.122189 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.122193 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.122197 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.122202 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.122206 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.122210 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.122214 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.122218 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.122222 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.122233 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.122237 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.122242 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.122246 | controller |       09:30:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.122250 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.122254 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.122259 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.122263 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.122267 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.122271 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.122275 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.122279 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.122284 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.122290 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.122312 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.122318 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.122322 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.122329 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.122335 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.122339 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.122343 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.122348 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.122352 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.122356 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.122360 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.122364 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.122369 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.122373 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.122377 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.122381 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.122385 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.122389 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.122394 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.122398 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.122402 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.122406 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.122410 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.122414 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.122419 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.122423 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.122427 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.122440 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.122445 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.122449 | controller |       09:30:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.122454 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.122458 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.122462 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.122466 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.122472 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.122477 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.122481 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.122485 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.122489 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.122493 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.122497 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.122502 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.122506 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.122510 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.122514 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.122518 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.122523 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.122527 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.122531 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.122535 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.122539 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.122543 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.122548 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.122552 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.122556 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.122560 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.122564 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.122568 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.122573 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.122577 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.122581 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.122585 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.122589 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.122593 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.122598 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.122602 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.122606 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.122612 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.122617 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.122628 | controller |       09:30:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.122632 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.122636 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.122641 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.122645 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.122649 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.122653 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.122657 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.122662 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.122666 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.122670 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.122674 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.122678 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.122682 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.122687 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.122691 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.122695 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.122700 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.122705 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.122711 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.122717 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.122723 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.122730 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.122736 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.122743 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.122750 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.122756 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.122763 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.122770 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.122777 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.122786 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.122793 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.122798 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.122803 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.122807 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.122811 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.122815 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.122820 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.122824 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.122828 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.122832 | controller |       09:30:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.122845 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.122850 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.122854 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.122859 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.122863 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.122867 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.122871 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.122875 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.122879 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.122884 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.122888 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.122892 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.122896 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.122903 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.122907 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.122913 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.122918 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.122922 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.122926 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.122930 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.122938 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.122942 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.122946 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.122950 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.122954 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.122959 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.122963 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.122967 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.122971 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.122975 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.122980 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.122984 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.122988 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.122992 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.122996 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.123001 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.123005 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.123009 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.123013 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.123017 | controller |       09:30:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.123022 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.123033 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.123037 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.123041 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.123045 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.123050 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.123054 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.123060 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.123064 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.123068 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.123072 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.123076 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.123081 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.123086 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.123091 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.123095 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.123099 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.123103 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.123107 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.123112 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.123116 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.123120 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.123124 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.123128 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.123132 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.123137 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.123141 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.123145 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.123149 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.123153 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.123157 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.123162 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.123166 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.123170 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.123174 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.123180 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.123184 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.123188 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.123192 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.123197 | controller |       09:30:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.123201 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.123205 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.123209 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.123220 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.123226 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.123230 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.123234 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.123238 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.123243 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.123247 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.123251 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.123255 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.123259 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.123263 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.123267 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.123272 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.123276 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.123280 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.123284 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.123302 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.123310 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.123314 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.123318 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.123323 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.123327 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.123331 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.123335 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.123339 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.123344 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.123348 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.123355 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.123359 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.123364 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.123368 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.123372 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.123376 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.123380 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.123386 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.123390 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.123395 | controller |       09:30:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.123399 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.123403 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.123407 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.123411 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.123423 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.123428 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.123432 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.123436 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.123440 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.123445 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.123451 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.123457 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.123466 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.123472 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.123477 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.123483 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.123488 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.123494 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.123499 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.123505 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.123511 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.123517 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.123523 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.123530 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.123536 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.123541 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.123545 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.123549 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.123553 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.123560 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.123565 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.123569 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.123573 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.123578 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.123582 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.123586 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.123590 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.123595 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.123599 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.123603 | controller |       09:30:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.123607 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.123612 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.123616 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.123620 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.123624 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.123629 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.123642 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.123646 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.123651 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.123655 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.123659 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.123664 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.123668 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.123672 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.123676 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.123680 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.123685 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.123689 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.123693 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.123697 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.123704 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.123709 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.123713 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.123717 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.123722 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.123726 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.123730 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.123734 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.123739 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.123743 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.123747 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.123751 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.123758 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.123763 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.123767 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.123771 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.123776 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.123780 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.123784 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.123788 | controller |       09:30:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.123793 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.123797 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.123801 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.123805 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.123811 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.123815 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.123820 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.123831 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.123837 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.123843 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.123850 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.123856 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.123863 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.123873 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.123881 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.123888 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.123895 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.123903 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.123910 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.123917 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.123926 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.123932 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.123936 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.123940 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.123945 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.123949 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.123953 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.123957 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.123962 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.123966 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.123970 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.123974 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.123979 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.123983 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.123987 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.123991 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.123996 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.124000 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.124004 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.124009 | controller |       09:30:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.124014 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.124018 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.124022 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.124026 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.124033 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.124039 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.124043 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.124047 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.124061 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.124066 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.124071 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.124075 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.124080 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.124084 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.124088 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.124092 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.124099 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.124104 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.124108 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.124112 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.124116 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.124121 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.124125 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.124129 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.124133 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.124138 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.124142 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.124147 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.124152 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.124156 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.124160 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.124164 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.124169 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.124174 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.124178 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.124182 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.124187 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.124193 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.124198 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.124203 | controller |       09:30:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.124207 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.124212 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.124216 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.124220 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.124224 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.124229 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.124233 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.124237 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.124242 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.124246 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.124257 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.124261 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.124266 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.124270 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.124275 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.124281 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.124286 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.124308 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.124313 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.124317 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.124321 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.124325 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.124330 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.124334 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.124338 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.124342 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.124347 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.124351 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.124357 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.124365 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.124369 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.124374 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.124378 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.124382 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.124387 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.124391 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.124395 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.124399 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.124403 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.124408 | controller |       09:30:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.124412 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.124416 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.124421 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.124425 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.124429 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.124433 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.124438 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.124442 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.124446 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.124450 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.124455 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.124466 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.124472 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.124476 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.124481 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.124485 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.124489 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.124494 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.124498 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.124502 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.124508 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.124512 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.124516 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.124521 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.124525 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.124529 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.124533 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.124538 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.124542 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.124546 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.124550 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.124555 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.124559 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.124563 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.124567 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.124572 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.124576 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.124580 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.124584 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.124589 | controller |       09:30:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.124593 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.124597 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.124601 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.124606 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.124610 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.124614 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.124618 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.124623 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.124627 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.124631 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.124637 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.124642 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.124646 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.124660 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.124664 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.124669 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.124673 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.124677 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.124682 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.124686 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.124690 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.124694 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.124699 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.124703 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.124707 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.124711 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.124716 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.124720 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.124724 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.124728 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.124734 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.124739 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.124743 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.124747 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.124752 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.124756 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.124760 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.124764 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.124769 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.124774 | controller |       09:30:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.124778 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.124783 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.124787 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.124791 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.124797 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.124801 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.124805 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.124810 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.124814 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.124818 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.124823 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.124827 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.124831 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.124836 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.124847 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.124852 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.124856 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.124860 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.124865 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.124869 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.124875 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.124879 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.124883 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.124888 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.124892 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.124896 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.124900 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.124907 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.124911 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.124915 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.124920 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.124924 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.124928 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.124979 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.124989 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.125022 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.125037 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.125049 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.125057 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.125063 | controller |       09:30:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.125068 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.125072 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.125077 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.125081 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.125085 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.125089 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.125094 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.125098 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.125102 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.125106 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.125111 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.125115 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.125119 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.125124 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.125128 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.125133 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.125147 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.125153 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.125157 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.125162 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.125166 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.125170 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.125174 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.125179 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.125183 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.125187 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.125191 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.125196 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.125201 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.125207 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.125211 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.125215 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.125220 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.125224 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.125228 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.125232 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.125237 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.125241 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.125245 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.125249 | controller |       09:30:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.125254 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.125258 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.125262 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.125266 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.125273 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.125278 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.125282 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.125289 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.125309 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.125315 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.125320 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.125325 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.125329 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.125333 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.125338 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.125342 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.125346 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.125358 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.125363 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.125367 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.125374 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.125378 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.125382 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.125387 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.125391 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.125395 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.125399 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.125403 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.125408 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.125413 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.125417 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.125421 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.125425 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.125430 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.125434 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.125438 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.125442 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.125447 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.125451 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.125455 | controller |       09:30:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.125460 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.125464 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.125468 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.125472 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.125476 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.125481 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.125485 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.125489 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.125493 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.125498 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.125502 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.125506 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.125510 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.125539 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.125543 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.125548 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.125552 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.125556 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.125567 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.125572 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.125576 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.125580 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.125585 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.125589 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.125593 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.125599 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.125603 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.125608 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.125612 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.125616 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.125620 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.125624 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.125629 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.125633 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.125637 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.125641 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.125645 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.125650 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.125654 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.125658 | controller |       09:30:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.125662 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.125666 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.125670 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.125675 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.125682 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.125686 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.125690 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.125694 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.125698 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.125703 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.125707 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.125711 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.125715 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.125719 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.125723 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.125729 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.125733 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.125737 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.125742 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.125752 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.125757 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.125761 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.125765 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.125770 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.125774 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.125778 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.125782 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.125787 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.125791 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.125795 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.125799 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.125803 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.125808 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.125812 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.125816 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.125820 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.125824 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.125830 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.125834 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.125839 | controller |       09:30:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.125843 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.125847 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.125851 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.125855 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.125860 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.125864 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.125868 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.125872 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.125878 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.125882 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.125886 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.125890 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.125895 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.125899 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.125903 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.125907 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.125911 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.125915 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.125920 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.125924 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.125929 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.125940 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.125944 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.125948 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.125953 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.125957 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.125961 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.125966 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.125970 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.125976 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.125980 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.125984 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.125988 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.125993 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.125997 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.126001 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.126005 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.126009 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.126013 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.126018 | controller |       09:30:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.126022 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.126026 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.126030 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.126035 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.126039 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.126043 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.126047 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.126051 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.126055 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.126060 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.126064 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.126068 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.126072 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.126076 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.126081 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.126085 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.126089 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.126093 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.126099 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.126104 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.126113 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.126119 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.126138 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.126147 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.126157 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.126164 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.126172 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.126179 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.126185 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.126191 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.126199 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.126203 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.126207 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.126211 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.126216 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.126220 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.126224 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.126228 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.126232 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.126236 | controller |       09:30:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.126241 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.126245 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.126249 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.126253 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.126257 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.126264 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.126268 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.126272 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.126277 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.126281 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.126285 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.126305 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.126309 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.126316 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.126320 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.126324 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.126328 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.126333 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.126337 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.126341 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.126345 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.126349 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.126354 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.126358 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.126371 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.126375 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.126380 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.126384 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.126388 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.126392 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.126397 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.126401 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.126405 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.126409 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.126413 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.126418 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.126422 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.126426 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.126433 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.126437 | controller |       09:30:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.126442 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.126446 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.126450 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.126455 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.126461 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.126465 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.126470 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.126474 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.126478 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.126482 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.126486 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.126491 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.126495 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.126499 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.126503 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.126507 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.126512 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.126516 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.126520 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.126524 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.126528 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.126533 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.126537 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.126541 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.126545 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.126556 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.126560 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.126565 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.126569 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.126573 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.126578 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.126582 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.126586 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.126590 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.126595 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.126599 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.126603 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.126609 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.126613 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.126617 | controller |       09:30:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.126621 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.126626 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.126630 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.126634 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.126642 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.126646 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.126650 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.126654 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.126659 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.126663 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.126667 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.126671 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.126675 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.126679 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.126684 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.126688 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.126692 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.126696 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.126700 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.126705 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.126709 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.126713 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.126717 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.126723 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.126727 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.126731 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.126735 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.126746 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.126750 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.126756 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.126760 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.126765 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.126769 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.126773 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.126778 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.126782 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.126786 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.126790 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.126794 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.126799 | controller |       09:30:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.126803 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.126807 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.126811 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.126815 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.126820 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.126824 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.126828 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.126832 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.126836 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.126841 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.126845 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.126849 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.126853 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.126858 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.126862 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.126866 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.126870 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.126874 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.126879 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.126883 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.126890 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.126894 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.126898 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.126903 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.126907 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.126911 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.126915 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.126919 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.126930 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.126934 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.126938 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.126943 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.126947 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.126951 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.126955 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.126960 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.126964 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.126968 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.126972 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.126976 | controller |       09:30:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.126981 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.126985 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.126991 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.126995 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.126999 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.127003 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.127007 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.127012 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.127016 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.127020 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.127024 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.127028 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.127032 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.127038 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.127042 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.127048 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.127052 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.127056 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.127060 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.127065 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.127069 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.127073 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.127077 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.127081 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.127086 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.127090 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.127094 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.127098 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.127102 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.127113 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.127117 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.127121 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.127126 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.127130 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.127134 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.127139 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.127143 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.127147 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.127151 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.127155 | controller |       09:30:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.127160 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.127164 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.127168 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.127172 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.127178 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.127182 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.127186 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.127190 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.127195 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.127199 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.127203 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.127207 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.127211 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.127215 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.127220 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.127224 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.127229 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.127234 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.127240 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.127246 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.127252 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.127261 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.127268 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.127275 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.127281 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.127304 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.127315 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.127322 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.127328 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.127332 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.127336 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.127350 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.127354 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.127358 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.127363 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.127367 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.127371 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.127379 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.127383 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.127387 | controller |       09:30:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.127391 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.127396 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.127400 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.127404 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.127408 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.127412 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.127417 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.127421 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.127425 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.127429 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.127433 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.127438 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.127442 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.127446 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.127450 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.127454 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.127459 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.127463 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.127467 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.127471 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.127475 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.127480 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.127484 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.127488 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.127492 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.127496 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.127501 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.127505 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.127509 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.127515 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.127521 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.127525 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.127536 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.127540 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.127545 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.127549 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.127553 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.127558 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.127562 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.127566 | controller |       09:30:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.127570 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.127576 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.127580 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.127585 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.127589 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.127593 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.127597 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.127601 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.127606 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.127610 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.127614 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.127618 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.127622 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.127626 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.127631 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.127635 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.127639 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.127643 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.127647 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.127651 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.127657 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.127661 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.127665 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.127670 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.127674 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.127678 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.127682 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.127686 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.127690 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.127695 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.127699 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.127703 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.127707 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.127718 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.127723 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.127727 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.127731 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.127735 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.127740 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.127744 | controller |       09:30:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.127748 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:30:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.127752 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.127757 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.127761 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.127765 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.127769 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.127773 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.127777 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.127782 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.127786 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.127790 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.127794 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.127799 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.127808 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.127812 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.127816 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.127822 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.127827 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.127831 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.127835 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.127840 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.127845 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.127849 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.127853 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.127857 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.127862 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.127866 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.127870 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.127874 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.127878 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.127882 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.127887 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.127891 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.127895 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.127906 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.127910 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.127915 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.127919 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:30:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.127923 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.127927 | controller |       09:30:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.127932 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.127936 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.127940 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.127944 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.127951 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.127956 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.127960 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.127964 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.127968 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.127972 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.127976 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.127981 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.127985 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.127989 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.127993 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.127997 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.128002 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.128006 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.128010 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.128014 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.128018 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.128023 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.128027 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.128031 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.128035 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.128041 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.128046 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.128052 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.128056 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.128060 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.128064 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.128070 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.128074 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.128079 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.128083 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.128087 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.128101 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.128107 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.128111 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.128116 | controller |       09:31:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.128122 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.128127 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.128131 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.128135 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.128139 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.128144 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.128148 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.128152 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.128156 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.128160 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.128164 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.128169 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.128173 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.128177 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.128181 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.128185 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.128189 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.128194 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.128198 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.128202 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.128206 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.128210 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.128215 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.128219 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.128223 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.128227 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.128231 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.128235 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.128239 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.128245 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.128249 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.128254 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.128258 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.128262 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.128266 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.128270 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.128275 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.128302 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.128313 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.128317 | controller |       09:31:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.128322 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.128326 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.128331 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.128335 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.128339 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.128343 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.128347 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.128351 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.128356 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.128361 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.128366 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.128372 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.128378 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.128384 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.128393 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.128402 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.128410 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.128417 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.128424 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.128431 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.128441 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.128448 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.128454 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.128460 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.128466 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.128472 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.128477 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.128481 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.128485 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.128489 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.128493 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.128498 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.128502 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.128506 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.128510 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.128515 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.128519 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.128523 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.128527 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.128541 | controller |       09:31:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.128546 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.128550 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.128554 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.128559 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.128563 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.128570 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.128574 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.128579 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.128592 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.128597 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.128601 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.128605 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.128609 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.128616 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.128620 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.128624 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.128628 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.128633 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.128637 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.128641 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.128645 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.128649 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.128654 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.128658 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.128662 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.128666 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.128670 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.128675 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.128679 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.128683 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.128687 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.128691 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.128695 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.128700 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.128704 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.128708 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.128712 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.128717 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.128722 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.128727 | controller |       09:31:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.128738 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.128743 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.128747 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.128751 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.128758 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.128763 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.128767 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.128771 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.128775 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.128779 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.128784 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.128788 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.128792 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.128796 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.128800 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.128805 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.128809 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.128813 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.128817 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.128821 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.128827 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.128832 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.128836 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.128840 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.128844 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.128848 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.128853 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.128857 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.128861 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.128865 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.128871 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.128875 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.128879 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.128883 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.128888 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.128892 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.128896 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.128902 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.128906 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.128911 | controller |       09:31:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.128915 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.128926 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.128930 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.128964 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.128969 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.128973 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.128977 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.128981 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.128986 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.128990 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.128994 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.128998 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.129003 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.129009 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.129014 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.129018 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.129022 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.129027 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.129031 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.129035 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.129039 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.129044 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.129048 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.129052 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.129056 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.129061 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.129065 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.129069 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.129073 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.129079 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.129083 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.129087 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.129092 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.129096 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.129100 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.129104 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.129108 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.129113 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.129117 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.129121 | controller |       09:31:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.129126 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.129130 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.129134 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.129146 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.129150 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.129155 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.129159 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.129163 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.129168 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.129172 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.129176 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.129180 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.129185 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.129189 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.129193 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.129197 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.129201 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.129206 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.129210 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.129214 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.129220 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.129224 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.129228 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.129232 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.129237 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.129241 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.129245 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.129249 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.129254 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.129258 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.129262 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.129266 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.129270 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.129275 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.129279 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.129285 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.129304 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.129311 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.129315 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.129320 | controller |       09:31:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.129324 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.129328 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.129332 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.129337 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.129351 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.129355 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.129360 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.129364 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.129368 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.129372 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.129377 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.129381 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.129385 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.129391 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.129395 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.129399 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.129403 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.129408 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.129412 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.129416 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.129420 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.129425 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.129454 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.129459 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.129463 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.129468 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.129472 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.129476 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.129480 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.129484 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.129488 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.129494 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.129499 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.129505 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.129511 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.129517 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.129535 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.129541 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.129547 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.129554 | controller |       09:31:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.129561 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.129568 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.129575 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.129581 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.129592 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.129613 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.129620 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.129625 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.129629 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.129633 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.129637 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.129642 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.129649 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.129653 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.129657 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.129661 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.129666 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.129670 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.129674 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.129678 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.129683 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.129687 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.129691 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.129695 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.129699 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.129704 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.129708 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.129712 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.129727 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.129731 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.129735 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.129739 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.129744 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.129748 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.129752 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.129756 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.129761 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.129767 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.129771 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.129775 | controller |       09:31:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.129779 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.129783 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.129788 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.129792 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.129796 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.129800 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.129804 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.129817 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.129822 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.129826 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.129831 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.129835 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.129839 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.129843 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.129847 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.129853 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.129858 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.129862 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.129866 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.129871 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.129881 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.129885 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.129889 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.129894 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.129898 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.129902 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.129906 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.129910 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.129914 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.129922 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.129926 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.129930 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.129936 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.129940 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.129945 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.129949 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.129953 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.129957 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.129961 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.129966 | controller |       09:31:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.129970 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.129974 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.129978 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.129982 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.129986 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.129991 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.129995 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.129999 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.130011 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.130015 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.130020 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.130024 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.130028 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.130032 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.130037 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.130041 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.130045 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.130049 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.130054 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.130058 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.130064 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.130068 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.130072 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.130076 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.130081 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.130085 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.130089 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.130093 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.130098 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.130102 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.130106 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.130110 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.130114 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.130119 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.130123 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.130127 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.130131 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.130135 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.130140 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.130144 | controller |       09:31:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.130148 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.130152 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.130157 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.130161 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.130165 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.130169 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.130173 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.130178 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.130182 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.130186 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.130197 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.130203 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.130207 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.130213 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.130217 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.130221 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.130225 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.130230 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.130234 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.130238 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.130242 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.130246 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.130251 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.130255 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.130259 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.130264 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.130268 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.130272 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.130276 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.130281 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.130286 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.130307 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.130312 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.130316 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.130320 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.130324 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.130329 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.130333 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.130337 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.130341 | controller |       09:31:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.130345 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.130349 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.130354 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.130358 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.130364 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.130368 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.130372 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.130376 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.130381 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.130385 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.130389 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.130400 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.130405 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.130409 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.130413 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.130418 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.130422 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.130426 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.130431 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.130435 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.130439 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.130443 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.130448 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.130452 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.130458 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.130463 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.130469 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.130475 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.130481 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.130486 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.130490 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.130494 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.130498 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.130502 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.130507 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.130511 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.130515 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.130521 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.130525 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.130529 | controller |       09:31:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.130533 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.130538 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.130542 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.130546 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.130550 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.130554 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.130559 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.130563 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.130567 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.130571 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.130575 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.130580 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.130584 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.130596 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.130600 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.130605 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.130609 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.130613 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.130617 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.130622 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.130626 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.130630 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.130634 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.130640 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.130644 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.130649 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.130654 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.130660 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.130666 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.130675 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.130681 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.130687 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.130694 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.130701 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.130707 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.130714 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.130720 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.130727 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.130734 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.130741 | controller |       09:31:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.130747 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.130751 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.130756 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.130760 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.130767 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.130771 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.130775 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.130780 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.130784 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.130788 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.130794 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.130798 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.130802 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.130806 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.130820 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.130825 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.130829 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.130834 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.130838 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.130842 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.130850 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.130854 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.130858 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.130863 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.130867 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.130871 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.130875 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.130879 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.130883 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.130888 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.130892 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.130896 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.130900 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.130904 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.130909 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.130913 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.130917 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.130921 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.130928 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.130932 | controller |       09:31:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.130937 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.130942 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.130946 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.130950 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.130954 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.130958 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.130963 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.130967 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.130971 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.130975 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.130979 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.130983 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.130988 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.130993 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.130998 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.131002 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.131013 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.131018 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.131022 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.131026 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.131031 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.131035 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.131039 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.131043 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.131047 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.131052 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.131056 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.131062 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.131066 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.131070 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.131074 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.131078 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.131083 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.131087 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.131091 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.131095 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.131099 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.131104 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.131108 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.131112 | controller |       09:31:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.131116 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.131120 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.131125 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.131129 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.131135 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.131139 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.131143 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.131147 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.131151 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.131156 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.131160 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.131164 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.131168 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.131172 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.131177 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.131182 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.131186 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.131197 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.131202 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.131206 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.131210 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.131215 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.131219 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.131223 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.131227 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.131231 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.131236 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.131240 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.131244 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.131248 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.131253 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.131257 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.131261 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.131265 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.131269 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.131274 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.131278 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.131284 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.131302 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.131310 | controller |       09:31:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.131315 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.131319 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.131323 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.131327 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.131331 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.131336 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.131340 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.131344 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.131348 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.131354 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.131358 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.131362 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.131367 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.131371 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.131375 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.131379 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.131383 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.131388 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.131399 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.131405 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.131409 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.131413 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.131417 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.131421 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.131426 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.131430 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.131434 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.131438 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.131443 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.131450 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.131456 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.131462 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.131468 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.131474 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.131480 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.131486 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.131491 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.131495 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.131499 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.131504 | controller |       09:31:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.131508 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.131512 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.131516 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.131520 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.131525 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.131529 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.131533 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.131537 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.131541 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.131545 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.131550 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.131554 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.131558 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.131581 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.131587 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.131591 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.131595 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.131599 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.131603 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.131614 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.131621 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.131625 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.131629 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.131633 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.131638 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.131644 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.131648 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.131652 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.131656 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.131661 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.131666 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.131670 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.131675 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.131679 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.131683 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.131687 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.131692 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.131696 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.131700 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.131704 | controller |       09:31:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.131708 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.131713 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.131717 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.131721 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.131725 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.131729 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.131733 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.131738 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.131742 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.131746 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.131750 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.131754 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.131758 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.131765 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.131769 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.131773 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.131777 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.131782 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.131787 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.131792 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.131808 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.131816 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.131824 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.131831 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.131837 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.131844 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.131851 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.131858 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.131865 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.131871 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.131877 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.131883 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.131889 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.131894 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.131900 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.131906 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.131911 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.131917 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.131922 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.131927 | controller |       09:31:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.131932 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.131936 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.131940 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.131944 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.131951 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.131955 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.131959 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.131963 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.131970 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.131974 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.131978 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.131982 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.131986 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.131991 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.131995 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.131999 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.132003 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.132007 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.132011 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.132016 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.132020 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.132024 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.132038 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.132042 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.132047 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.132051 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.132055 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.132059 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.132064 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.132068 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.132072 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.132076 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.132080 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.132085 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.132089 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.132093 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.132097 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.132104 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.132108 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.132113 | controller |       09:31:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.132117 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.132121 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.132125 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.132129 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.132135 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.132139 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.132143 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.132148 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.132152 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.132156 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.132160 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.132164 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.132169 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.132173 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.132177 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.132181 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.132185 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.132189 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.132194 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.132198 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.132202 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.132206 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.132210 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.132222 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.132228 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.132232 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.132237 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.132241 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.132245 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.132251 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.132255 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.132259 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.132263 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.132267 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.132272 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.132276 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.132280 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.132284 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.132309 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.132316 | controller |       09:31:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.132321 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.132325 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.132329 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.132333 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.132337 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.132342 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.132346 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.132350 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.132354 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.132358 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.132362 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.132366 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.132371 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.132375 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.132379 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.132383 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.132387 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.132391 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.132395 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.132400 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.132406 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.132410 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.132414 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.132418 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.132422 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.132434 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.132439 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.132443 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.132447 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.132451 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.132456 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.132460 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.132464 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.132468 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.132473 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.132478 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.132483 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.132490 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.132496 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.132502 | controller |       09:31:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.132507 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.132511 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.132515 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.132522 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.132526 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.132530 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.132534 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.132539 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.132543 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.132547 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.132551 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.132555 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.132559 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.132565 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.132570 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.132575 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.132579 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.132584 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.132588 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.132592 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.132596 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.132600 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.132605 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.132609 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.132613 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.132617 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.132631 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.132637 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.132643 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.132648 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.132652 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.132657 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.132661 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.132665 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.132669 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.132673 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.132678 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.132682 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.132686 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.132690 | controller |       09:31:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.132694 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.132699 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.132703 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.132707 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.132713 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.132717 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.132721 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.132726 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.132730 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.132734 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.132738 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.132742 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.132746 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.132751 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.132755 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.132759 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.132763 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.132767 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.132772 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.132776 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.132782 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.132786 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.132790 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.132796 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.132801 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.132805 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.132809 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.132813 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.132824 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.132829 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.132833 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.132837 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.132842 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.132846 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.132850 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.132854 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.132858 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.132864 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.132868 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.132873 | controller |       09:31:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.132877 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.132881 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.132885 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.132889 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.132894 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.132898 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.132902 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.132906 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.132910 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.132915 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.132921 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.132927 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.132933 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.132939 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.132945 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.132951 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.132958 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.132965 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.132971 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.132978 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.132985 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.132992 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.132999 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.133006 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.133012 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.133017 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.133021 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.133026 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.133030 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.133047 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.133054 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.133059 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.133064 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.133068 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.133073 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.133077 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.133081 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.133085 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.133090 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.133094 | controller |       09:31:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.133098 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.133103 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.133109 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.133113 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.133118 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.133125 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.133129 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.133133 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.133138 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.133142 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.133146 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.133151 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.133155 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.133159 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.133163 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.133168 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.133172 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.133176 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.133180 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.133185 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.133191 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.133195 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.133199 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.133204 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.133208 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.133212 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.133216 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.133221 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.133225 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.133229 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.133233 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.133244 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.133248 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.133253 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.133257 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.133261 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.133266 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.133270 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.133274 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.133279 | controller |       09:31:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.133283 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.133290 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.133313 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.133318 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.133322 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.133326 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.133331 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.133335 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.133339 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.133344 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.133348 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.133352 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.133356 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.133363 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.133368 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.133372 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.133376 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.133381 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.133385 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.133389 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.133394 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.133400 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.133405 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.133409 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.133413 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.133417 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.133422 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.133426 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.133430 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.133434 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.133439 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.133443 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.133454 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.133458 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.133463 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.133467 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.133471 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.133475 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.133480 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.133484 | controller |       09:31:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.133488 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.133493 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.133497 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.133501 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.133509 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.133513 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.133518 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.133522 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.133526 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.133546 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.133552 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.133556 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.133560 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.133602 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.133608 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.133613 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.133617 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.133621 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.133625 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.133629 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.133634 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.133638 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.133642 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.133646 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.133650 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.133654 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.133659 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.133663 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.133667 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.133674 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.133678 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.133682 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.133686 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.133697 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.133702 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.133706 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.133710 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.133716 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.133720 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.133725 | controller |       09:31:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.133729 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.133734 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.133739 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.133743 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.133747 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.133751 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.133755 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.133760 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.133764 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.133768 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.133772 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.133776 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.133780 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.133785 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.133789 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.133793 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.133797 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.133801 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.133806 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.133810 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.133816 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.133820 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.133824 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.133828 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.133833 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.133837 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.133841 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.133845 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.133850 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.133855 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.133859 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.133864 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.133868 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.133872 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.133882 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.133887 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.133892 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.133896 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.133900 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.133905 | controller |       09:31:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.133909 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.133913 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.133917 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.133921 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.133926 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.133930 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.133934 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.133938 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.133942 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.133946 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.133951 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.133955 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.133959 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.133963 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.133967 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.133973 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.133979 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.133983 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.134012 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.134017 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.134024 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.134028 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.134032 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.134036 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.134040 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.134045 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.134050 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.134056 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.134062 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.134068 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.134074 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.134080 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.134086 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.134093 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.134100 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.134106 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.134124 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.134132 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.134139 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.134146 | controller |       09:31:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.134152 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.134156 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.134161 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.134165 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.134169 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.134173 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.134177 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.134181 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.134186 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.134190 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.134194 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.134198 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.134202 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.134209 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.134213 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.134217 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.134222 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.134226 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.134230 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.134234 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.134238 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.134243 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.134247 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.134251 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.134255 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.134259 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.134263 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.134268 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.134272 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.134276 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.134280 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.134284 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.134303 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.134310 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.134315 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.134319 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.134323 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.134335 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.134340 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.134344 | controller |       09:31:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.134352 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.134357 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.134361 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.134365 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.134371 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.134375 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.134379 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.134384 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.134388 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.134392 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.134396 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.134400 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.134404 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.134409 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.134413 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.134417 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.134421 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.134425 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.134430 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.134434 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.134438 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.134442 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.134446 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.134450 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.134455 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.134459 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.134463 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.134467 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.134471 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.134476 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.134481 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.134485 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.134490 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.134494 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.134498 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.134502 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.134506 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.134512 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.134516 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.134528 | controller |       09:31:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.134533 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.134537 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.134542 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.134546 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.134550 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.134555 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.134559 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.134563 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.134567 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.134572 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.134576 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.134580 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.134584 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.134589 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.134594 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.134599 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.134603 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.134607 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.134611 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.134616 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.134620 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.134624 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.134628 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.134632 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.134637 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.134641 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.134645 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.134649 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.134653 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.134659 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.134663 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.134668 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.134672 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.134676 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.134680 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.134685 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.134689 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.134693 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.134697 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.134701 | controller |       09:31:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.134712 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.134717 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.134721 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.134725 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.134729 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.134734 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.134738 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.134742 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.134746 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.134751 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.134755 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.134759 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.134763 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.134768 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.134772 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.134776 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.134780 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.134784 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.134789 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.134793 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.134802 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.134808 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.134814 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.134820 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.134824 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.134828 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.134832 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.134837 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.134841 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.134845 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.134849 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.134853 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.134857 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.134862 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.134866 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.134870 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.134874 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.134878 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.134885 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.134889 | controller |       09:31:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.134894 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.134906 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.134910 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.134915 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.134923 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.134927 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.134932 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.134937 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.134942 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.134947 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.134951 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.134955 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.134959 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.134965 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.134969 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.134973 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.134978 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.134982 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.134986 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.134990 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.134994 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.134998 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.135003 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.135007 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.135011 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.135015 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.135019 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.135023 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.135028 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.135032 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.135036 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.135040 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.135044 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.135049 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.135053 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.135057 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.135061 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.135065 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.135069 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.135074 | controller |       09:31:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.135078 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.135082 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.135086 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.135097 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.135103 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.135107 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.135112 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.135116 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.135120 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.135124 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.135128 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.135133 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.135137 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.135142 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.135147 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.135151 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.135155 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.135159 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.135163 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.135168 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.135172 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.135176 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.135180 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.135185 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.135190 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.135196 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.135202 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.135208 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.135214 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.135220 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.135227 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.135234 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.135241 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.135247 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.135255 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.135262 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.135267 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.135275 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.135281 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.135303 | controller |       09:31:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.135312 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.135318 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.135324 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.135330 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.135346 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.135352 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.135358 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.135364 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.135370 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.135376 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.135382 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.135388 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.135394 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.135400 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.135405 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.135415 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.135421 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.135427 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.135433 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.135440 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.135446 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.135452 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.135458 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.135464 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.135471 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.135477 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.135484 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.135491 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.135498 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.135509 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.135514 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.135519 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.135523 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.135527 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.135531 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.135538 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.135542 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.135546 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.135552 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.135556 | controller |       09:31:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.135560 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.135564 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.135569 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.135573 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.135577 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.135591 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.135596 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.135600 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.135604 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.135609 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.135613 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.135617 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.135621 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.135626 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.135630 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.135634 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.135638 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.135642 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.135647 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.135651 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.135657 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.135661 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.135665 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.135669 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.135674 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.135678 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.135682 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.135686 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.135690 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.135694 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.135698 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.135703 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.135707 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.135711 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.135715 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.135719 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.135723 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.135728 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.135732 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.135736 | controller |       09:31:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.135740 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.135744 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.135749 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.135753 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.135757 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.135761 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.135765 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.135777 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.135782 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.135786 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.135791 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.135795 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.135801 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.135807 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.135811 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.135815 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.135819 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.135824 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.135828 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.135832 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.135837 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.135842 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.135846 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.135850 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.135854 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.135858 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.135863 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.135867 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.135871 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.135875 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.135881 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.135885 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.135889 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.135893 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.135898 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.135904 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.135910 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.135916 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.135922 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.135929 | controller |       09:31:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.135937 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.135942 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.135949 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.135956 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.135964 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.135969 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.135973 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.135978 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.135991 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.135996 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.136000 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.136004 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.136009 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.136013 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.136017 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.136022 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.136026 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.136030 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.136034 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.136039 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.136043 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.136047 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.136051 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.136056 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.136060 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.136064 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.136068 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.136073 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.136077 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.136081 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.136086 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.136090 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.136096 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.136101 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.136105 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.136109 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.136113 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.136119 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.136123 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.136128 | controller |       09:31:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.136132 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.136136 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.136140 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.136145 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.136149 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.136153 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.136157 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.136161 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.136166 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.136170 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.136182 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.136186 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.136190 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.136195 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.136199 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.136203 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.136207 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.136211 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.136216 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.136220 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.136224 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.136228 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.136233 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.136237 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.136241 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.136245 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.136249 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.136253 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.136258 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.136263 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.136267 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.136272 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.136276 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.136280 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.136284 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.136304 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.136311 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.136315 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.136320 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.136324 | controller |       09:31:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.136328 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.136332 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.136336 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.136341 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.136347 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.136351 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.136355 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.136359 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.136364 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.136368 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.136372 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.136386 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.136390 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.136395 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.136399 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.136403 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.136407 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.136412 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.136416 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.136420 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.136427 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.136432 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.136436 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.136440 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.136445 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.136450 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.136456 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.136462 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.136468 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.136474 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.136480 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.136486 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.136492 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.136496 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.136500 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.136504 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.136509 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.136513 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.136517 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.136521 | controller |       09:31:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.136526 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.136530 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.136534 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.136538 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.136542 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.136547 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.136553 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.136559 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.136565 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.136571 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.136577 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.136584 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.136603 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.136614 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.136622 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.136629 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.136635 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.136641 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.136647 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.136651 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.136656 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.136660 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.136664 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.136668 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.136672 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.136676 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.136681 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.136685 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.136691 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.136696 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.136700 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.136704 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.136708 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.136713 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.136717 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.136721 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.136725 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.136729 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.136733 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.136738 | controller |       09:31:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.136742 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.136746 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.136750 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.136754 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.136760 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.136765 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.136769 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.136773 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.136777 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.136781 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.136785 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.136790 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.136794 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.136798 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.136810 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.136816 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.136821 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.136825 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.136829 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.136834 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.136839 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.136844 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.136848 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.136852 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.136856 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.136895 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.136900 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.136904 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.136909 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.136913 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.136917 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.136921 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.136926 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.136930 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.136934 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.136938 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.136943 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.136950 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.136954 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.136958 | controller |       09:31:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.136963 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.136967 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.136971 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.136975 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.136979 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.136984 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.136988 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.136992 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.136996 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.137000 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.137007 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.137011 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.137016 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.137020 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.137024 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.137036 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.137040 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.137045 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.137049 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.137053 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.137058 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.137062 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.137066 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.137071 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.137075 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.137079 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.137083 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.137088 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.137092 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.137097 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.137102 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.137106 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.137110 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.137114 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.137119 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.137123 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.137127 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.137131 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.137135 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.137140 | controller |       09:31:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.137144 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.137148 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.137153 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.137157 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.137161 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.137165 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.137169 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.137174 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.137178 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.137182 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.137186 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.137190 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.137195 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.137199 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.137203 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.137207 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.137211 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.137221 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.137225 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.137230 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.137235 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.137240 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.137244 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.137248 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.137252 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.137257 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.137261 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.137267 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.137271 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.137275 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.137280 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.137285 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.137305 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.137313 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.137317 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.137321 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.137325 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.137330 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.137334 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.137338 | controller |       09:31:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.137342 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.137346 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.137351 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.137355 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.137359 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.137363 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.137367 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.137371 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.137376 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.137380 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.137384 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.137388 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.137392 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.137398 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.137402 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.137406 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.137411 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.137415 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.137425 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.137429 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.137434 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.137438 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.137442 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.137447 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.137451 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.137455 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.137459 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.137463 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.137467 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.137471 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.137476 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.137480 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.137484 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.137488 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.137492 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.137497 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.137501 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.137505 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.137509 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.137513 | controller |       09:31:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.137517 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.137522 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.137526 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.137530 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.137536 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.137567 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.137572 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.137577 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.137581 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.137587 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.137591 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.137595 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.137600 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.137604 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.137608 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.137612 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.137616 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.137621 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.137625 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.137634 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.137639 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.137643 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.137647 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.137652 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.137656 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.137660 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.137664 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.137669 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.137673 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.137679 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.137684 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.137690 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.137697 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.137703 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.137709 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.137716 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.137723 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.137735 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.137742 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.137749 | controller |       09:31:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.137757 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.137763 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.137769 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.137774 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.137781 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.137787 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.137791 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.137795 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.137799 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.137804 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.137808 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.137812 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.137816 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.137820 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.137824 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.137829 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.137833 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.137837 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.137841 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.137845 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.137858 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.137864 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.137868 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.137873 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.137877 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.137881 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.137885 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.137891 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.137895 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.137901 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.137906 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.137910 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.137915 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.137920 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.137925 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.137929 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.137934 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.137939 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.137944 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.137948 | controller |       09:31:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.137952 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.137957 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.137961 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.137965 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.137969 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.137973 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.137977 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.137982 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.137986 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.137990 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.137994 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.137998 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.138002 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.138007 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.138011 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.138015 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.138019 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.138023 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.138028 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.138032 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.138040 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.138044 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.138054 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.138058 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.138062 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.138066 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.138071 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.138075 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.138079 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.138083 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.138088 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.138092 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.138096 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.138100 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.138104 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.138108 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.138113 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.138117 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.138121 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.138125 | controller |       09:31:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.138130 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:31:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.138136 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.138141 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.138146 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.138151 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.138156 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.138162 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.138167 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.138174 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.138180 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.138185 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.138191 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.138196 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.138204 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.138210 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.138217 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.138223 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.138228 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.138234 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.138240 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.138246 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.138251 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.138257 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.138271 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.138277 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.138283 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.138305 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.138314 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.138320 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.138325 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.138331 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.138335 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.138339 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.138343 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.138348 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.138352 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.138356 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.138360 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:31:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.138364 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.138371 | controller |       09:31:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.138375 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.138379 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.138384 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.138388 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.138395 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.138401 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.138406 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.138412 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.138417 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.138423 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.138429 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.138434 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.138440 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.138445 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.138450 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.138454 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.138459 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.138463 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.138467 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.138471 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.138475 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.138479 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.138484 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.138488 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.138494 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.138509 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.138515 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.138521 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.138527 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.138532 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.138537 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.138543 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.138549 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.138555 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.138560 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.138566 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.138571 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.138579 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.138584 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.138589 | controller |       09:32:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.138594 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.138599 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.138603 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.138608 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.138614 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.138620 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.138626 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.138632 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.138637 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.138644 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.138650 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.138656 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.138661 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.138667 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.138673 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.138679 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.138684 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.138690 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.138696 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.138701 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.138707 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.138712 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.138718 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.138722 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.138726 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.138731 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.138744 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.138750 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.138756 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.138764 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.138775 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.138781 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.138786 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.138792 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.138797 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.138803 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.138808 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.138814 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.138821 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.138826 | controller |       09:32:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.138832 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.138837 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.138842 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.138850 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.138856 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.138861 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.138866 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.138872 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.138877 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.138882 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.138888 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.138893 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.138898 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.138903 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.138908 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.138913 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.138919 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.138924 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.138928 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.138933 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.138943 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.138949 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.138954 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.138959 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.138965 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.138971 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.138977 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.138983 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.139004 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.139013 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.139019 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.139026 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.139033 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.139039 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.139045 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.139051 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.139056 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.139062 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.139067 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.139072 | controller |       09:32:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.139078 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.139084 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.139091 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.139098 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.139104 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.139110 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.139115 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.139121 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.139127 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.139132 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.139138 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.139144 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.139149 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.139158 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.139163 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.139169 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.139175 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.139180 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.139186 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.139192 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.139197 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.139203 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.139208 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.139228 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.139233 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.139239 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.139244 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.139250 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.139255 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.139272 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.139278 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.139283 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.139306 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.139316 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.139323 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.139329 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.139335 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.139341 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.139348 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.139354 | controller |       09:32:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.139359 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.139365 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.139371 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.139376 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.139388 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.139394 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.139399 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.139405 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.139412 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.139418 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.139424 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.139430 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.139436 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.139442 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.139448 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.139453 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.139458 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.139464 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.139469 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.139475 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.139480 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.139485 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.139491 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.139496 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.139502 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.139507 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.139513 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.139519 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.139524 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.139529 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.139546 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.139551 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.139557 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.139562 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.139568 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.139574 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.139579 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.139588 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.139594 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.139599 | controller |       09:32:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.139605 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.139611 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.139619 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.139625 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.139630 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.139636 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.139641 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.139646 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.139652 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.139657 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.139661 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.139665 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.139669 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.139673 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.139677 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.139682 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.139686 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.139690 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.139694 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.139698 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.139702 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.139706 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.139711 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.139715 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.139719 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.139723 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.139727 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.139731 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.139735 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.139741 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.139746 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.139750 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.139762 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.139767 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.139771 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.139775 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.139779 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.139783 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.139788 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.139792 | controller |       09:32:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.139796 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.139800 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.139804 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.139808 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.139813 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.139817 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.139821 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.139825 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.139829 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.139833 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.139838 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.139842 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.139846 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.139850 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.139854 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.139860 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.139864 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.139868 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.139873 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.139877 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.139883 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.139888 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.139892 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.139896 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.139901 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.139905 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.139909 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.139913 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.139917 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.139921 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.139926 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.139930 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.139934 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.139945 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.139949 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.139953 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.139958 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.139962 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.139966 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.139970 | controller |       09:32:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.139975 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.139979 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.139983 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.139987 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.139991 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.139995 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.140000 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.140004 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.140008 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.140012 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.140016 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.140020 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.140025 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.140030 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.140034 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.140039 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.140043 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.140047 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.140051 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.140055 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.140062 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.140067 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.140072 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.140076 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.140080 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.140084 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.140088 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.140093 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.140097 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.140101 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.140105 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.140109 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.140113 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.140118 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.140130 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.140135 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.140139 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.140145 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.140150 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.140156 | controller |       09:32:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.140162 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.140171 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.140177 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.140183 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.140193 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.140200 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.140207 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.140214 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.140220 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.140227 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.140235 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.140241 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.140247 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.140253 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.140259 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.140264 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.140270 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.140276 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.140281 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.140290 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.140319 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.140323 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.140327 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.140332 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.140336 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.140340 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.140344 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.140348 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.140353 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.140357 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.140365 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.140369 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.140373 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.140378 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.140382 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.140386 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.140399 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.140407 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.140415 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.140421 | controller |       09:32:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.140427 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.140432 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.140438 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.140442 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.140446 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.140451 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.140455 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.140459 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.140464 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.140469 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.140474 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.140480 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.140486 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.140491 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.140497 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.140502 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.140510 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.140515 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.140521 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.140525 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.140530 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.140534 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.140538 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.140542 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.140546 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.140550 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.140555 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.140559 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.140563 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.140569 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.140573 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.140577 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.140582 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.140586 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.140590 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.140594 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.140600 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.140615 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.140621 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.140627 | controller |       09:32:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.140633 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.140639 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.140644 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.140650 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.140655 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.140660 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.140666 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.140671 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.140677 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.140682 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.140687 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.140693 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.140698 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.140704 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.140709 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.140714 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.140720 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.140725 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.140730 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.140736 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.140744 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.140750 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.140755 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.140759 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.140763 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.140768 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.140772 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.140777 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.140783 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.140789 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.140795 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.140801 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.140807 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.140813 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.140819 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.140824 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.140830 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.140835 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.140855 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.140862 | controller |       09:32:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.140869 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.140875 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.140880 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.140885 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.140891 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.140896 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.140900 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.140904 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.140908 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.140913 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.140918 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.140923 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.140927 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.140935 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.140940 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.140944 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.140948 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.140953 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.140957 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.140961 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.140965 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.140969 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.140974 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.140978 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.140982 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.140986 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.140990 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.140994 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.140998 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.141002 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.141007 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.141011 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.141015 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.141019 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.141023 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.141027 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.141032 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.141036 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.141040 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.141052 | controller |       09:32:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.141057 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.141061 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.141066 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.141070 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.141076 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.141080 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.141084 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.141088 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.141092 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.141097 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.141101 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.141105 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.141109 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.141113 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.141119 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.141123 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.141127 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.141131 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.141135 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.141140 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.141147 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.141153 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.141159 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.141164 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.141171 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.141177 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.141183 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.141188 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.141193 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.141197 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.141201 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.141206 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.141210 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.141214 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.141218 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.141222 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.141226 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.141234 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.141238 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.141242 | controller |       09:32:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.141247 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.141259 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.141263 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.141268 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.141272 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.141276 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.141280 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.141285 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.141419 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.141428 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.141432 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.141437 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.141441 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.141445 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.141449 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.141457 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.141461 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.141465 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.141469 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.141474 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.141478 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.141482 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.141486 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.141490 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.141494 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.141498 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.141503 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.141507 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.141511 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.141517 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.141521 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.141526 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.141530 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.141534 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.141538 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.141542 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.141546 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.141574 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.141581 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.141585 | controller |       09:32:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.141590 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.141594 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.141608 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.141613 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.141617 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.141621 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.141626 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.141630 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.141634 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.141638 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.141643 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.141647 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.141651 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.141655 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.141659 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.141663 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.141667 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.141672 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.141676 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.141680 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.141686 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.141690 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.141695 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.141699 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.141703 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.141707 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.141711 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.141715 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.141719 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.141724 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.141728 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.141732 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.141736 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.141740 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.141744 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.141749 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.141753 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.141757 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.141761 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.141765 | controller |       09:32:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.141769 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.141774 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.141778 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.141782 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.141793 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.141797 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.141802 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.141806 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.141810 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.141814 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.141819 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.141823 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.141827 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.141835 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.141839 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.141843 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.141847 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.141852 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.141856 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.141860 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.141864 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.141869 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.141873 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.141877 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.141881 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.141885 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.141890 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.141894 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.141898 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.141902 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.141908 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.141912 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.141916 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.141920 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.141925 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.141930 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.141934 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.141938 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.141943 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.141947 | controller |       09:32:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.141951 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.141955 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.141959 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.141963 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.141969 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.141980 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.141985 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.141989 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.141993 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.141998 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.142002 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.142006 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.142011 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.142016 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.142021 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.142027 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.142033 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.142039 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.142044 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.142050 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.142056 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.142061 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.142066 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.142071 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.142077 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.142082 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.142087 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.142093 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.142098 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.142104 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.142108 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.142112 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.142116 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.142120 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.142125 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.142131 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.142136 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.142142 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.142146 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.142150 | controller |       09:32:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.142154 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.142158 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.142162 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.142167 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.142171 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.142175 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.142179 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.142191 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.142196 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.142200 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.142205 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.142209 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.142213 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.142217 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.142221 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.142226 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.142230 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.142234 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.142238 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.142242 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.142248 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.142252 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.142256 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.142261 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.142265 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.142269 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.142273 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.142277 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.142281 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.142300 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.142310 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.142316 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.142322 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.142327 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.142333 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.142338 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.142344 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.142349 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.142354 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.142359 | controller |       09:32:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.142363 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.142367 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.142371 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.142375 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.142381 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.142386 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.142390 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.142394 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.142406 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.142411 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.142416 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.142422 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.142430 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.142437 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.142442 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.142448 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.142455 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.142461 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.142467 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.142474 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.142485 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.142493 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.142500 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.142506 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.142513 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.142520 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.142526 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.142532 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.142538 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.142544 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.142550 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.142555 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.142561 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.142567 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.142571 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.142575 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.142579 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.142584 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.142588 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.142592 | controller |       09:32:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.142596 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.142611 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.142616 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.142620 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.142624 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.142629 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.142633 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.142637 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.142641 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.142655 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.142660 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.142664 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.142669 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.142675 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.142679 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.142683 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.142687 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.142692 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.142696 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.142700 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.142704 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.142708 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.142713 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.142717 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.142721 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.142725 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.142729 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.142733 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.142738 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.142742 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.142746 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.142750 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.142756 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.142760 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.142764 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.142769 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.142773 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.142777 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.142781 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.142785 | controller |       09:32:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.142790 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.142794 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.142798 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.142802 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.142808 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.142814 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.142818 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.142822 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.142826 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.142830 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.142835 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.142846 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.142850 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.142854 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.142859 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.142863 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.142869 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.142873 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.142877 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.142881 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.142886 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.142890 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.142894 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.142898 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.142902 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.142907 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.142911 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.142915 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.142919 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.142923 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.142927 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.142931 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.142936 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.142940 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.142944 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.142948 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.142952 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.142958 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.142962 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.142967 | controller |       09:32:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.142971 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.142975 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.142979 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.142983 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.142988 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.142992 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.142996 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.143000 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.143004 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.143008 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.143013 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.143018 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.143029 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.143034 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.143038 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.143042 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.143046 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.143050 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.143055 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.143059 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.143063 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.143067 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.143071 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.143075 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.143080 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.143084 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.143088 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.143092 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.143096 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.143103 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.143108 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.143112 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.143116 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.143120 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.143125 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.143129 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.143133 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.143137 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.143141 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.143145 | controller |       09:32:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.143150 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.143154 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.143158 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.143162 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.143166 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.143171 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.143175 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.143179 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.143183 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.143187 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.143191 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.143196 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.143200 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.143204 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.143215 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.143219 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.143223 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.143228 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.143232 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.143237 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.143244 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.143249 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.143253 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.143257 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.143261 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.143265 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.143270 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.143274 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.143279 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.143283 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.143304 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.143313 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.143317 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.143321 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.143325 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.143330 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.143334 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.143338 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.143342 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.143348 | controller |       09:32:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.143353 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.143359 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.143365 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.143370 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.143376 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.143382 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.143388 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.143393 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.143399 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.143405 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.143410 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.143415 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.143421 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.143430 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.143435 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.143448 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.143452 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.143456 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.143461 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.143465 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.143469 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.143473 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.143478 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.143482 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.143486 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.143490 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.143494 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.143499 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.143503 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.143507 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.143511 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.143515 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.143519 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.143524 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.143528 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.143532 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.143536 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.143540 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.143545 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.143549 | controller |       09:32:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.143553 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.143557 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.143561 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.143566 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.143573 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.143579 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.143585 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.143591 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.143597 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.143604 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.143614 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.143621 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.143628 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.143635 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.143642 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.143648 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.143656 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.143673 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.143678 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.143683 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.143687 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.143691 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.143695 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.143700 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.143704 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.143708 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.143712 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.143716 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.143720 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.143725 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.143729 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.143733 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.143737 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.143741 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.143745 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.143750 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.143754 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.143760 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.143765 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.143769 | controller |       09:32:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.143773 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.143777 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.143781 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.143786 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.143792 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.143796 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.143801 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.143805 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.143809 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.143813 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.143817 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.143822 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.143826 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.143830 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.143834 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.143838 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.143842 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.143847 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.143858 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.143863 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.143867 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.143872 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.143876 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.143882 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.143886 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.143890 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.143895 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.143900 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.143904 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.143910 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.143914 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.143919 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.143923 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.143927 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.143931 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.143935 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.143940 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.143944 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.143948 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.143952 | controller |       09:32:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.143957 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.143961 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.143965 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.143969 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.143973 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.143978 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.143982 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.143986 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.143990 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.143994 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.143999 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.144003 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.144007 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.144011 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.144015 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.144019 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.144024 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.144028 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.144032 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.144043 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.144048 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.144053 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.144057 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.144061 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.144065 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.144070 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.144074 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.144078 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.144082 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.144086 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.144091 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.144095 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.144099 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.144103 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.144107 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.144111 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.144116 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.144120 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.144124 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.144128 | controller |       09:32:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.144132 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.144137 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.144141 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.144145 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.144149 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.144153 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.144157 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.144161 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.144166 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.144171 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.144175 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.144180 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.144184 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.144189 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.144193 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.144199 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.144203 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.144207 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.144212 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.144216 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.144228 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.144232 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.144237 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.144241 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.144245 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.144249 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.144254 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.144258 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.144262 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.144266 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.144270 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.144274 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.144279 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.144283 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.144290 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.144309 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.144314 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.144318 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.144322 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.144326 | controller |       09:32:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.144330 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.144334 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.144339 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.144343 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.144349 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.144353 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.144357 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.144362 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.144366 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.144370 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.144374 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.144378 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.144383 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.144387 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.144391 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.144395 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.144399 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.144403 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.144408 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.144412 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.144416 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.144420 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.144432 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.144436 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.144440 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.144446 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.144451 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.144455 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.144459 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.144463 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.144467 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.144472 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.144476 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.144480 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.144484 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.144488 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.144492 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.144499 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.144504 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.144508 | controller |       09:32:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.144512 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.144516 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.144520 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.144525 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.144529 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.144533 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.144537 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.144541 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.144546 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.144550 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.144554 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.144558 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.144562 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.144566 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.144571 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.144575 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.144579 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.144583 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.144587 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.144593 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.144597 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.144601 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.144605 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.144616 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.144621 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.144625 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.144629 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.144634 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.144638 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.144644 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.144649 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.144654 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.144658 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.144662 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.144666 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.144670 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.144675 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.144679 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.144683 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.144687 | controller |       09:32:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.144691 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.144696 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.144700 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.144705 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.144710 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.144716 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.144722 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.144728 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.144737 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.144744 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.144750 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.144757 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.144764 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.144771 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.144778 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.144784 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.144792 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.144798 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.144804 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.144809 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.144815 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.144819 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.144824 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.144828 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.144832 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.144846 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.144851 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.144856 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.144860 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.144864 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.144868 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.144902 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.144907 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.144911 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.144916 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.144920 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.144924 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.144928 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.144932 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.144937 | controller |       09:32:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.144942 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.144946 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.144950 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.144954 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.144959 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.144963 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.144967 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.144971 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.144975 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.144979 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.144984 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.144988 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.144992 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.144998 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.145002 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.145006 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.145011 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.145015 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.145019 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.145023 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.145027 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.145032 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.145036 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.145040 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.145047 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.145051 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.145065 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.145071 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.145077 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.145083 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.145088 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.145094 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.145099 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.145105 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.145110 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.145115 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.145120 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.145125 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.145133 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.145138 | controller |       09:32:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.145143 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.145149 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.145154 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.145160 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.145169 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.145173 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.145177 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.145181 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.145185 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.145189 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.145194 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.145198 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.145202 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.145206 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.145210 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.145214 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.145219 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.145223 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.145227 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.145231 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.145237 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.145241 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.145246 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.145250 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.145254 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.145258 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.145262 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.145275 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.145279 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.145284 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.145304 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.145312 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.145316 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.145320 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.145325 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.145329 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.145333 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.145339 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.145344 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.145348 | controller |       09:32:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.145354 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.145359 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.145365 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.145372 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.145378 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.145383 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.145389 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.145394 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.145399 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.145405 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.145410 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.145416 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.145422 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.145427 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.145432 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.145438 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.145442 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.145446 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.145450 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.145456 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.145462 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.145468 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.145474 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.145479 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.145485 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.145491 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.145498 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.145504 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.145509 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.145526 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.145531 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.145535 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.145540 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.145544 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.145548 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.145552 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.145571 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.145578 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.145583 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.145588 | controller |       09:32:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.145593 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.145598 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.145603 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.145608 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.145613 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.145618 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.145623 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.145628 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.145633 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.145638 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.145643 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.145647 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.145652 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.145657 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.145662 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.145670 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.145675 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.145680 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.145685 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.145690 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.145698 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.145703 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.145708 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.145714 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.145719 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.145724 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.145729 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.145734 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.145739 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.145744 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.145757 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.145763 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.145768 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.145773 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.145778 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.145783 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.145788 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.145793 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.145798 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.145803 | controller |       09:32:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.145808 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.145813 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.145817 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.145821 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.145826 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.145830 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.145834 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.145838 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.145844 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.145850 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.145856 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.145862 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.145868 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.145878 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.145885 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.145893 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.145900 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.145907 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.145914 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.145922 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.145929 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.145935 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.145941 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.145946 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.145950 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.145954 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.145959 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.145963 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.145967 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.145971 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.145975 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.145989 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.145994 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.145998 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.146002 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.146007 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.146011 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.146015 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.146019 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.146023 | controller |       09:32:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.146028 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.146032 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.146039 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.146043 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.146049 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.146053 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.146058 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.146062 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.146066 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.146070 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.146074 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.146078 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.146083 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.146087 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.146091 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.146095 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.146099 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.146103 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.146108 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.146112 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.146116 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.146120 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.146124 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.146129 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.146133 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.146137 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.146141 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.146145 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.146149 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.146154 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.146159 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.146164 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.146168 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.146179 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.146184 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.146188 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.146193 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.146198 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.146203 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.146207 | controller |       09:32:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.146211 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.146215 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.146219 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.146224 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.146228 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.146232 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.146236 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.146240 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.146245 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.146249 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.146253 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.146257 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.146261 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.146266 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.146270 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.146274 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.146278 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.146282 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.146286 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.146311 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.146324 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.146330 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.146334 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.146339 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.146343 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.146347 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.146351 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.146355 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.146360 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.146366 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.146370 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.146374 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.146378 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.146382 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.146394 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.146398 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.146403 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.146407 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.146411 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.146416 | controller |       09:32:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.146420 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.146424 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.146428 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.146432 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.146436 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.146441 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.146445 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.146449 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.146453 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.146457 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.146461 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.146466 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.146470 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.146474 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.146478 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.146485 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.146489 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.146493 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.146498 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.146502 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.146507 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.146512 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.146516 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.146520 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.146524 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.146529 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.146533 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.146537 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.146541 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.146545 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.146550 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.146554 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.146558 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.146562 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.146566 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.146577 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.146582 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.146586 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.146590 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.146595 | controller |       09:32:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.146599 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.146605 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.146609 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.146613 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.146619 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.146623 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.146627 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.146632 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.146636 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.146640 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.146644 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.146648 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.146652 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.146658 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.146663 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.146667 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.146671 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.146675 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.146680 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.146685 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.146691 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.146696 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.146702 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.146708 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.146714 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.146718 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.146722 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.146726 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.146730 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.146735 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.146739 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.146743 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.146747 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.146751 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.146756 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.146760 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.146764 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.146776 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.146780 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.146784 | controller |       09:32:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.146789 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.146793 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.146797 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.146802 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.146809 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.146813 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.146818 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.146822 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.146826 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.146830 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.146834 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.146839 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.146843 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.146847 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.146851 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.146855 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.146861 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.146866 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.146870 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.146874 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.146878 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.146883 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.146887 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.146891 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.146895 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.146899 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.146904 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.146908 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.146912 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.146916 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.146920 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.146924 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.146929 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.146933 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.146937 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.146941 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.146945 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.146951 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.146962 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.146967 | controller |       09:32:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.146971 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.146976 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.146981 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.146987 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.146993 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.147000 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.147006 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.147012 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.147019 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.147026 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.147032 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.147039 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.147046 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.147052 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.147059 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.147069 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.147075 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.147079 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.147084 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.147088 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.147092 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.147096 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.147100 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.147105 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.147109 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.147113 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.147117 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.147121 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.147125 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.147132 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.147137 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.147141 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.147145 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.147149 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.147153 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.147157 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.147162 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.147166 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.147170 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.147185 | controller |       09:32:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.147191 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.147195 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.147200 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.147204 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.147208 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.147212 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.147216 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.147221 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.147225 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.147229 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.147233 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.147237 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.147241 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.147245 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.147250 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.147254 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.147258 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.147262 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.147266 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.147271 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.147278 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.147283 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.147290 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.147310 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.147314 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.147318 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.147322 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.147326 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.147331 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.147335 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.147339 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.147343 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.147347 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.147352 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.147356 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.147360 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.147364 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.147368 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.147372 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.147377 | controller |       09:32:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.147381 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.147393 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.147398 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.147402 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.147407 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.147413 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.147417 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.147422 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.147426 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.147430 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.147434 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.147439 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.147443 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.147450 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.147454 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.147459 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.147465 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.147469 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.147474 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.147478 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.147482 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.147486 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.147491 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.147495 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.147499 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.147503 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.147508 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.147512 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.147516 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.147520 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.147525 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.147529 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.147535 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.147539 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.147544 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.147548 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.147552 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.147557 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.147561 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.147565 | controller |       09:32:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.147569 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.147574 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.147585 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.147590 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.147597 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.147601 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.147605 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.147610 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.147614 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.147618 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.147622 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.147627 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.147631 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.147635 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.147639 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.147644 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.147648 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.147652 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.147656 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.147661 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.147665 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.147669 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.147674 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.147678 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.147682 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.147686 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.147691 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.147695 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.147699 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.147703 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.147707 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.147712 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.147716 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.147720 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.147725 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.147729 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.147733 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.147740 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.147746 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.147750 | controller |       09:32:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.147756 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.147760 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.147764 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.147768 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.147779 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.147784 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.147788 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.147793 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.147797 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.147801 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.147806 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.147810 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.147814 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.147818 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.147823 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.147827 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.147831 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.147835 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.147840 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.147844 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.147848 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.147852 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.147856 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.147861 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.147865 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.147869 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.147873 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.147878 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.147882 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.147890 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.147895 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.147899 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.147903 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.147908 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.147912 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.147916 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.147920 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.147925 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.147929 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.147933 | controller |       09:32:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.147938 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.147942 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.147947 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.147951 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.147957 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.147969 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.147974 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.147978 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.147983 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.147987 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.147991 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.147996 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.148000 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.148007 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.148011 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.148015 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.148020 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.148024 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.148028 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.148033 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.148039 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.148043 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.148047 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.148051 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.148056 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.148060 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.148064 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.148068 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.148073 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.148077 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.148081 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.148085 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.148090 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.148094 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.148098 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.148103 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.148107 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.148113 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.148119 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.148125 | controller |       09:32:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.148131 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.148137 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.148144 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.148151 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.148158 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.148166 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.148185 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.148194 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.148202 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.148208 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.148214 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.148220 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.148226 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.148235 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.148241 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.148246 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.148252 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.148258 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.148311 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.148322 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.148330 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.148334 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.148338 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.148343 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.148347 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.148351 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.148356 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.148360 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.148364 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.148368 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.148373 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.148377 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.148381 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.148385 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.148390 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.148396 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.148400 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.148404 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.148408 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.148413 | controller |       09:32:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.148418 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.148422 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.148426 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.148430 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.148436 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.148441 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.148445 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.148449 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.148463 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.148468 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.148472 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.148477 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.148481 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.148485 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.148489 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.148495 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.148499 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.148504 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.148508 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.148512 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.148516 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.148521 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.148525 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.148529 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.148533 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.148538 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.148542 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.148546 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.148550 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.148554 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.148559 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.148563 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.148568 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.148573 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.148577 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.148581 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.148585 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.148591 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.148596 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.148600 | controller |       09:32:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.148605 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.148609 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.148614 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.148618 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.148622 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.148626 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.148631 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.148635 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.148639 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.148650 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.148655 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.148659 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.148671 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.148675 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.148679 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.148684 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.148688 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.148692 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.148696 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.148701 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.148705 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.148709 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.148713 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.148718 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.148722 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.148727 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.148731 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.148736 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.148740 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.148750 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.148755 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.148759 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.148763 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.148768 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.148772 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.148776 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.148781 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.148785 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.148789 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.148793 | controller |       09:32:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.148798 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.148802 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.148806 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.148811 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.148815 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.148819 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.148823 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.148828 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.148832 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.148836 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.148840 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.148852 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.148857 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.148861 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.148865 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.148870 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.148874 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.148878 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.148883 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.148887 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.148894 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.148898 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.148903 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.148907 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.148911 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.148916 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.148920 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.148924 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.148928 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.148933 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.148939 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.148944 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.148949 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.148954 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.148958 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.148962 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.148966 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.148971 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.148975 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.148979 | controller |       09:32:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.148984 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.148988 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.148992 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.148996 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.149001 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.149005 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.149009 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.149017 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.149022 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.149026 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.149030 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.149034 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.149045 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.149052 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.149056 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.149061 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.149065 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.149069 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.149074 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.149078 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.149082 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.149087 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.149091 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.149095 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.149099 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.149104 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.149108 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.149112 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.149116 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.149121 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.149125 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.149129 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.149133 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.149138 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.149142 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.149146 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.149150 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.149155 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.149159 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.149163 | controller |       09:32:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.149168 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:32:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.149172 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.149176 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.149180 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.149187 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.149191 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.149196 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.149200 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.149204 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.149208 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.149213 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.149219 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.149224 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.149228 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.149239 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.149244 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.149250 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.149256 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.149262 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.149268 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.149277 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.149284 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.149356 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.149362 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.149368 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.149372 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.149376 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.149381 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.149385 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.149389 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.149393 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.149397 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.149402 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.149406 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.149410 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.149414 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.149419 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.149425 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:32:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.149430 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.149434 | controller |       09:32:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.149438 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.149442 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.149448 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.149454 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.149464 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.149470 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.149476 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.149482 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.149488 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.149492 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.149497 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.149501 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.149505 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.149509 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.149513 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.149527 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.149532 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.149536 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.149540 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.149545 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.149549 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.149553 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.149557 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.149563 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.149586 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.149591 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.149595 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.149599 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.149606 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.149613 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.149619 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.149624 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.149630 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.149636 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.149642 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.149648 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.149653 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.149659 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.149665 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.149671 | controller |       09:33:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.149676 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.149682 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.149687 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.149692 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.149697 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.149703 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.149707 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.149711 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.149715 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.149719 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.149724 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.149728 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.149732 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.149736 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.149740 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.149744 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.149749 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.149762 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.149767 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.149771 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.149777 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.149782 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.149786 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.149790 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.149794 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.149798 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.149802 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.149807 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.149811 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.149815 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.149819 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.149823 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.149828 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.149832 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.149836 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.149840 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.149844 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.149848 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.149855 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.149859 | controller |       09:33:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.149863 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.149867 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.149872 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.149876 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.149880 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.149884 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.149888 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.149893 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.149897 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.149901 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.149907 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.149911 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.149915 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.149921 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.149925 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.149931 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.149935 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.149939 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.149950 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.149954 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.149959 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.149963 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.149967 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.149971 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.149976 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.149980 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.149984 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.149988 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.149992 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.149997 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.150001 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.150005 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.150009 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.150013 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.150018 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.150022 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.150026 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.150030 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.150034 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.150039 | controller |       09:33:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.150043 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.150047 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.150051 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.150055 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.150061 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.150065 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.150070 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.150074 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.150078 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.150082 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.150086 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.150091 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.150095 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.150099 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.150103 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.150107 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.150111 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.150116 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.150120 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.150130 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.150135 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.150139 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.150144 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.150148 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.150153 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.150157 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.150161 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.150168 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.150172 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.150177 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.150181 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.150185 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.150189 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.150193 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.150198 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.150202 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.150206 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.150212 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.150216 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.150220 | controller |       09:33:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.150225 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.150229 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.150233 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.150237 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.150241 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.150245 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.150250 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.150254 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.150258 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.150262 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.150266 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.150271 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.150275 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.150279 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.150283 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.150301 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.150308 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.150313 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.150317 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.150321 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.150335 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.150340 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.150344 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.150349 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.150353 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.150357 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.150361 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.150365 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.150369 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.150375 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.150382 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.150388 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.150394 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.150400 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.150406 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.150412 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.150419 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.150425 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.150432 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.150439 | controller |       09:33:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.150445 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.150452 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.150458 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.150464 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.150471 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.150477 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.150484 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.150491 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.150497 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.150506 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.150512 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.150517 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.150523 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.150528 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.150534 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.150540 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.150545 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.150550 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.150554 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.150558 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.150565 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.150579 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.150585 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.150589 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.150594 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.150598 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.150602 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.150606 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.150610 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.150614 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.150619 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.150623 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.150627 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.150631 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.150636 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.150640 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.150644 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.150648 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.150652 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.150656 | controller |       09:33:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.150661 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.150665 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.150669 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.150673 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.150677 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.150682 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.150686 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.150690 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.150694 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.150698 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.150702 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.150707 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.150711 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.150716 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.150721 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.150725 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.150729 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.150733 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.150737 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.150742 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.150746 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.150750 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.150754 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.150765 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.150770 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.150776 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.150780 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.150785 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.150789 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.150794 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.150798 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.150802 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.150806 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.150810 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.150815 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.150819 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.150823 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.150827 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.150831 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.150836 | controller |       09:33:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.150840 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.150844 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.150848 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.150852 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.150864 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.150868 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.150873 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.150877 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.150881 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.150885 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.150889 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.150893 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.150898 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.150902 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.150906 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.150910 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.150914 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.150918 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.150923 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.150927 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.150931 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.150935 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.150939 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.150943 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.150954 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.150960 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.150966 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.150972 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.150976 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.150980 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.150984 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.150988 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.150993 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.150997 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.151001 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.151005 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.151009 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.151015 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.151020 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.151024 | controller |       09:33:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.151028 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.151032 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.151037 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.151041 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.151045 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.151049 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.151053 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.151058 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.151063 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.151068 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.151072 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.151076 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.151080 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.151084 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.151089 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.151093 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.151097 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.151101 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.151105 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.151109 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.151114 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.151118 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.151122 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.151126 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.151130 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.151134 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.151145 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.151150 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.151154 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.151160 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.151165 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.151169 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.151173 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.151177 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.151181 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.151186 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.151190 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.151194 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.151198 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.151202 | controller |       09:33:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.151207 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.151211 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.151215 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.151219 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.151223 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.151227 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.151232 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.151236 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.151240 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.151244 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.151248 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.151252 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.151257 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.151261 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.151265 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.151271 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.151275 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.151279 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.151284 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.151302 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.151314 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.151319 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.151323 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.151327 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.151333 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.151337 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.151341 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.151353 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.151357 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.151362 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.151366 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.151371 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.151375 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.151380 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.151384 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.151389 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.151393 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.151397 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.151401 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.151406 | controller |       09:33:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.151410 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.151414 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.151418 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.151423 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.151427 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.151431 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.151435 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.151440 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.151444 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.151448 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.151454 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.151459 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.151465 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.151474 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.151480 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.151485 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.151491 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.151497 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.151502 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.151508 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.151514 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.151519 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.151525 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.151530 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.151535 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.151540 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.151546 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.151552 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.151558 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.151573 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.151579 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.151585 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.151591 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.151597 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.151602 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.151608 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.151613 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.151619 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.151624 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.151631 | controller |       09:33:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.151637 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.151643 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.151650 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.151660 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.151671 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.151679 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.151686 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.151693 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.151701 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.151708 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.151715 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.151721 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.151727 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.151733 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.151739 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.151744 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.151748 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.151752 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.151756 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.151761 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.151765 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.151769 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.151774 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.151779 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.151785 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.151791 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.151796 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.151802 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.151808 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.151814 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.151834 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.151840 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.151844 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.151849 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.151854 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.151860 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.151866 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.151874 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.151880 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.151886 | controller |       09:33:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.151891 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.151897 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.151902 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.151908 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.151913 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.151917 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.151921 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.151925 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.151930 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.151934 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.151938 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.151943 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.151947 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.151951 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.151955 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.151959 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.151964 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.151968 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.151972 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.151976 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.151981 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.151985 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.151989 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.151995 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.151999 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.152004 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.152008 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.152012 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.152016 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.152022 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.152027 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.152039 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.152044 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.152048 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.152053 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.152057 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.152061 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.152065 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.152070 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.152074 | controller |       09:33:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.152078 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.152082 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.152087 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.152091 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.152095 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.152101 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.152105 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.152109 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.152114 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.152118 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.152122 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.152126 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.152130 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.152134 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.152139 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.152143 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.152147 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.152151 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.152155 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.152160 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.152165 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.152172 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.152176 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.152180 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.152184 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.152189 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.152193 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.152197 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.152201 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.152205 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.152209 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.152214 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.152218 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.152229 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.152234 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.152238 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.152242 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.152246 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.152251 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.152255 | controller |       09:33:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.152259 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.152263 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.152269 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.152273 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.152279 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.152283 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.152290 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.152309 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.152314 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.152318 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.152322 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.152327 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.152331 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.152338 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.152342 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.152346 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.152351 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.152355 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.152359 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.152363 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.152369 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.152373 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.152377 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.152382 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.152386 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.152390 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.152394 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.152398 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.152403 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.152407 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.152411 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.152415 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.152419 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.152424 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.152435 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.152440 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.152444 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.152448 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.152453 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.152457 | controller |       09:33:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.152461 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.152465 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.152469 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.152474 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.152479 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.152483 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.152488 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.152492 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.152496 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.152500 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.152504 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.152508 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.152513 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.152517 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.152521 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.152525 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.152529 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.152534 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.152538 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.152542 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.152546 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.152552 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.152557 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.152561 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.152565 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.152569 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.152573 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.152577 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.152582 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.152586 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.152590 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.152594 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.152598 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.152603 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.152607 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.152618 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.152622 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.152629 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:17 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.152633 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.152637 | controller |       09:33:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.152641 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.152646 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.152650 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.152654 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.152658 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.152662 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.152666 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.152671 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.152675 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.152679 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.152683 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.152687 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.152692 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.152696 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.152700 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.152706 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.152710 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.152714 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.152718 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.152723 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.152727 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.152731 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.152735 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.152739 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.152743 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.152748 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.152752 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.152757 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.152763 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.152771 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.152777 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.152784 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.152790 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.152796 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.152803 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.152811 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.152817 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.152835 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.152844 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.152851 | controller |       09:33:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.152856 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.152863 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.152867 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.152871 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.152875 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.152880 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.152884 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.152888 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.152892 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.152896 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.152900 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.152905 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.152909 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.152913 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.152917 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.152921 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.152925 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.152930 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.152934 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.152938 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.152944 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.152948 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.152953 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.152957 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.152961 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.152965 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.152969 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.152973 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.152977 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.152982 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.152986 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.152990 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.152994 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.152999 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.153003 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.153007 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.153011 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.153015 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.153028 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.153032 | controller |       09:33:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.153036 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.153041 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.153045 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.153049 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.153053 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.153057 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.153061 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.153066 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.153070 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.153074 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.153078 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.153082 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.153086 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.153092 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.153096 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.153100 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.153106 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.153110 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.153115 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.153119 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.153123 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.153127 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.153131 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.153136 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.153140 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.153144 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.153148 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.153152 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.153157 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.153161 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.153166 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.153171 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.153175 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.153179 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.153183 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.153187 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.153191 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.153196 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.153200 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.153211 | controller |       09:33:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.153215 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.153219 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.153224 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.153228 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.153234 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.153238 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.153242 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.153246 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.153250 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.153255 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.153259 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.153263 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.153267 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.153271 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.153275 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.153280 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.153284 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.153301 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.153309 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.153314 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.153320 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.153324 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.153329 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.153333 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.153337 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.153341 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.153345 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.153350 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.153354 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.153358 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.153362 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.153366 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.153370 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.153375 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.153379 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.153383 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.153387 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.153393 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.153397 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.153401 | controller |       09:33:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.153408 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.153420 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.153425 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.153430 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.153434 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.153438 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.153442 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.153448 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.153453 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.153459 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.153466 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.153472 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.153478 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.153484 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.153490 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.153496 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.153502 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.153507 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.153512 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.153518 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.153523 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.153527 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.153531 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.153535 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.153539 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.153544 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.153548 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.153552 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.153556 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.153563 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.153597 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.153602 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.153606 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.153611 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.153615 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.153619 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.153623 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.153627 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.153631 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.153636 | controller |       09:33:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.153640 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.153644 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.153656 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.153661 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.153668 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.153673 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.153677 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.153681 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.153686 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.153690 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.153694 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.153698 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.153702 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.153706 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.153712 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.153716 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.153721 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.153725 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.153729 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.153733 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.153739 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.153744 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.153748 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.153752 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.153756 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.153760 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.153764 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.153769 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.153773 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.153777 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.153781 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.153785 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.153789 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.153794 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.153798 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.153802 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.153806 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.153810 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.153815 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.153819 | controller |       09:33:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.153823 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.153827 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.153831 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.153836 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.153847 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.153851 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.153855 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.153860 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.153864 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.153868 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.153872 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.153876 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.153881 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.153886 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.153892 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.153897 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.153903 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.153909 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.153915 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.153922 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.153928 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.153935 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.153942 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.153949 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.153956 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.153963 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.153970 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.153977 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.153982 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.153988 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.153994 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.153999 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.154005 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.154011 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.154016 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.154021 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.154027 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.154031 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.154038 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.154042 | controller |       09:33:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.154047 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.154051 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.154055 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.154059 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.154065 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.154080 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.154085 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.154089 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.154093 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.154098 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.154102 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.154106 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.154110 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.154114 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.154118 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.154124 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.154128 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.154133 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.154137 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.154141 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.154145 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.154149 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.154153 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.154158 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.154162 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.154166 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.154170 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.154174 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.154178 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.154183 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.154187 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.154192 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.154197 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.154201 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.154205 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.154209 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.154213 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.154219 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:26 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.154223 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.154227 | controller |       09:33:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.154232 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.154236 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.154240 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.154244 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.154248 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.154253 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.154264 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.154269 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.154274 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.154278 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.154282 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.154286 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.154307 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.154313 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.154318 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.154322 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.154326 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.154330 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.154334 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.154338 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.154344 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.154348 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.154352 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.154356 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.154361 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.154365 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.154369 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.154373 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.154377 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.154383 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.154388 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.154392 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.154396 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.154400 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.154404 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.154409 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.154413 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.154417 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.154423 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.154427 | controller |       09:33:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.154431 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.154435 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.154439 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.154443 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.154448 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.154452 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.154456 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.154460 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.154472 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.154477 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.154481 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.154485 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.154489 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.154494 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.154498 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.154502 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.154506 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.154510 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.154515 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.154519 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.154526 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.154530 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.154534 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.154539 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.154543 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.154547 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.154551 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.154555 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.154560 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.154564 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.154571 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.154577 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.154583 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.154589 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.154594 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.154601 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.154607 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.154612 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.154618 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.154623 | controller |       09:33:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.154629 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.154634 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.154640 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.154646 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.154651 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.154657 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.154663 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.154668 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.154674 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.154690 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.154696 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.154702 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.154707 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.154716 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.154722 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.154727 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.154733 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.154738 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.154744 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.154749 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.154755 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.154760 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.154766 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.154771 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.154777 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.154782 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.154788 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.154793 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.154799 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.154804 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.154810 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.154815 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.154821 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.154826 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.154832 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.154837 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.154843 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.154848 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.154853 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.154859 | controller |       09:33:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.154864 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.154870 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.154875 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.154881 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.154889 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.154895 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.154901 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.154906 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.154912 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.154917 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.154922 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.154940 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.154946 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.154952 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.154957 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.154963 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.154968 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.154973 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.154979 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.154984 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.154990 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.154995 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.155001 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.155006 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.155011 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.155017 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.155022 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.155028 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.155034 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.155039 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.155045 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.155050 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.155055 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.155061 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.155066 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.155072 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.155077 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.155085 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.155091 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.155096 | controller |       09:33:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.155102 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.155107 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.155112 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.155118 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.155125 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.155131 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.155136 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.155142 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.155147 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.155153 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.155158 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.155163 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.155178 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.155185 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.155191 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.155196 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.155202 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.155207 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.155213 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.155218 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.155224 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.155229 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.155234 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.155240 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.155245 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.155251 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.155256 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.155261 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.155267 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.155276 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.155281 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.155289 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.155314 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.155319 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.155326 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.155331 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.155336 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.155341 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.155348 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.155353 | controller |       09:33:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.155359 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.155364 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.155369 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.155374 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.155382 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.155387 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.155392 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.155398 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.155403 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.155408 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.155415 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.155421 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.155426 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.155444 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.155450 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.155456 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.155462 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.155468 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.155473 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.155479 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.155488 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.155493 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.155499 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.155504 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.155512 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.155517 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.155522 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.155527 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.155532 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.155538 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.155543 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.155548 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.155553 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.155559 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.155564 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.155569 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.155575 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.155580 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:33 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.155585 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.155590 | controller |       09:33:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.155596 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.155601 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.155606 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.155612 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.155617 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.155623 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.155628 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.155634 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.155639 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.155645 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.155650 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.155657 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.155663 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.155674 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.155680 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.155691 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.155697 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.155702 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.155707 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.155712 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.155718 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.155723 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.155728 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.155734 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.155739 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.155744 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.155750 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.155755 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.155760 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.155766 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.155771 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.155776 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.155782 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.155787 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.155792 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.155798 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.155803 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.155809 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.155814 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.155820 | controller |       09:33:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.155825 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.155830 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.155835 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.155841 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.155849 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.155855 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.155860 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.155865 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.155870 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.155876 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.155881 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.155886 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.155891 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.155897 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.155902 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.155907 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.155915 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.155921 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.155926 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.155932 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.155937 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.155942 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.155947 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.155953 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.155958 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.155963 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.155968 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.155974 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.155981 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.155987 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.155992 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.155997 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.156002 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.156008 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.156013 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.156018 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.156024 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.156031 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.156036 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.156041 | controller |       09:33:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.156047 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.156052 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.156057 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.156063 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.156068 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.156073 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.156078 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.156084 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.156089 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.156094 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.156099 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.156105 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.156110 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.156115 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.156120 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.156126 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.156131 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.156136 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.156145 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.156150 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.156155 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.156161 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.156166 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.156171 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.156176 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.156181 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.156186 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.156191 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.156196 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.156204 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.156211 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.156216 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.156222 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.156227 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.156233 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.156238 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.156243 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.156248 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.156253 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.156258 | controller |       09:33:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.156264 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.156269 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.156274 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.156279 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.156285 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.156307 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.156312 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.156318 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.156323 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.156328 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.156336 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.156341 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.156347 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.156352 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.156357 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.156362 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.156367 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.156372 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.156378 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.156392 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.156400 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.156406 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.156411 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.156416 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.156422 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.156427 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.156432 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.156438 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.156443 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.156448 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.156453 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.156459 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.156464 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.156469 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.156474 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.156480 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.156485 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.156490 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.156495 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.156501 | controller |       09:33:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.156506 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.156511 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.156516 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.156521 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.156527 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.156532 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.156537 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.156542 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.156548 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.156553 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.156559 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.156564 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.156570 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.156577 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.156583 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.156588 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.156594 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.156599 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.156605 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.156611 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.157215 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.157332 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.157354 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.157360 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.157365 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.157370 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.157375 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.157383 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.157388 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.157393 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.157398 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.157404 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.157409 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.157414 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.157419 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.157425 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.157430 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.157435 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.157440 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.157446 | controller |       09:33:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.157452 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.157457 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.157462 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.157467 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.157479 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.157485 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.157490 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.157496 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.157501 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.157506 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.157511 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.157516 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.157522 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.157527 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.157532 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.157537 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.157542 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.157548 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.157553 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.157558 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.157563 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.157608 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.157615 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.157620 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.157625 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.157630 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.157635 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.157640 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.157645 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.157650 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.157656 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.157661 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.157666 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.157671 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.157676 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.157681 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.157687 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.157694 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:40 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.157699 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.157704 | controller |       09:33:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.157709 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.157715 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.157720 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.157725 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.157730 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.157738 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.157743 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.157748 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.157754 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.157759 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.157764 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.157770 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.157775 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.157781 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.157786 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.157791 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.157797 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.157802 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.157807 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.157812 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.157821 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.157826 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.157831 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.157853 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.157859 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.157864 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.157869 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.157874 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.157879 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.157887 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.157892 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.157897 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.157903 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.157908 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.157913 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.157918 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.157924 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.157929 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.157934 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.157939 | controller |       09:33:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.157945 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.157950 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.157955 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.157960 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.157965 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.157971 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.157976 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.157981 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.157986 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.157991 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.157996 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.158001 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.158006 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.158011 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.158016 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.158023 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.158028 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.158033 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.158039 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.158044 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.158051 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.158056 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.158062 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.158066 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.158077 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.158084 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.158090 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.158095 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.158101 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.158115 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.158120 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.158126 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.158130 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.158135 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.158140 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.158145 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.158151 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.158155 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.158160 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.158165 | controller |       09:33:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.158170 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.158175 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.158181 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.158186 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.158191 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.158196 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.158201 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.158206 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.158211 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.158216 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.158221 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.158226 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.158231 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.158244 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.158250 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.158255 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.158260 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.158265 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.158270 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.158275 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.158280 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.158285 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.158303 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.158311 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.158316 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.158322 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.158337 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.158343 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.158348 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.158353 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.158359 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.158364 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.158369 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.158376 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.158382 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.158386 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.158392 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.158397 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.158408 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.158414 | controller |       09:33:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.158419 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.158424 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.158429 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.158434 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.158442 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.158447 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.158452 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.158457 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.158465 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.158470 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.158475 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.158480 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.158485 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.158490 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.158495 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.158500 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.158505 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.158510 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.158515 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.158520 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.158527 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.158532 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.158537 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.158542 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.158547 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.158552 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.158558 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.158566 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.158571 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.158577 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.158585 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.158590 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.158595 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.158600 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.158605 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.158611 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.158617 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.158625 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.158630 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.158636 | controller |       09:33:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.158641 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.158646 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.158651 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.158656 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.158661 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.158666 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.158671 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.158676 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.158682 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.158687 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.158692 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.158697 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.158702 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.158707 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.158712 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.158717 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.158722 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.158727 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.158732 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.158737 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.158743 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.158748 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.158753 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.158758 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.158765 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.158770 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.158775 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.158780 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.158785 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.158807 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.158813 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.158820 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.158825 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.158830 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.158835 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.158840 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.158845 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.158850 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.158855 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.158860 | controller |       09:33:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.158866 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.158872 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.158876 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.158881 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.158889 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.158894 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.158899 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.158904 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.158909 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.158914 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.158919 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.158925 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.158930 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.158935 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.158940 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.158945 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.158950 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.158955 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.158960 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.158965 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.158972 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.158977 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.158983 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.158988 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.158993 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.158998 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.159003 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.159008 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.159013 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.159018 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.159026 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.159031 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.159036 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.159041 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.159046 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.159051 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.159056 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.159062 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.159067 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.159072 | controller |       09:33:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.159077 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.159082 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.159087 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.159095 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.159116 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.159121 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.159126 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.159131 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.159136 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.159142 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.159146 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.159152 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.159157 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.159163 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.159168 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.159173 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.159178 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.159183 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.159188 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.159193 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.159198 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.159203 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.159208 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.159213 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.159218 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.159223 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.159228 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.159233 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.159238 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.159243 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.159248 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.159256 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.159261 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.159266 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.159271 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.159276 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.159281 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.159289 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:48 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.159311 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.159317 | controller |       09:33:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.159322 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.159327 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.159333 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.159338 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.159346 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.159351 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.159356 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.159361 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.159367 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.159372 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.159377 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.159382 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.159387 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.159393 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.159398 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.159403 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.159408 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.159413 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.159418 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.159424 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.159429 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.159434 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.159438 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.159446 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.159451 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.159456 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.159461 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.159466 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.159471 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.159476 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.159481 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.159486 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.159491 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.159506 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.159512 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.159517 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.159522 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.159529 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.159534 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.159539 | controller |       09:33:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.159544 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.159550 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.159555 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.159560 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.159565 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.159570 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.159575 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.159581 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.159586 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.159591 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.159596 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.159601 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.159616 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.159622 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.159627 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.159634 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.159639 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.159644 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.159649 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.159654 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.159661 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.159666 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.159671 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.159682 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.159687 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.159693 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.159699 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.159704 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.159709 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.159721 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.159727 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.159732 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.159738 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.159750 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.159758 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.159763 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.159768 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.159773 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.159778 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.159783 | controller |       09:33:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.159788 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.159793 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.159800 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.159812 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.159817 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.159822 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.159828 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.159833 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.159837 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.159842 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.159848 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.159853 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.159859 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.159864 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.159870 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.159875 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.159880 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.159885 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.159890 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.159895 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.159902 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.159907 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.159912 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.159917 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.159923 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.159928 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.159933 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.159938 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.159949 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.159954 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.159970 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.159975 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.159980 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.159985 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.159990 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.159997 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.160003 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.160008 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.160013 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.160018 | controller |       09:33:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.160023 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.160029 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.160034 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.160039 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.160044 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.160049 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.160054 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.160059 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.160064 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.160069 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.160074 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.160079 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.160084 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.160091 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.160096 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.160101 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.160106 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.160111 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.160117 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.160122 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.160127 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.160134 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.160140 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.160145 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.160150 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.160155 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.160160 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.160165 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.160170 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.160175 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.160182 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.160187 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.160192 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.160197 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.160203 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.160208 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.160213 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.160221 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.160226 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.160231 | controller |       09:33:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.160236 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.160241 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.160247 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.160252 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.160259 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.160264 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.160269 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.160274 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.160280 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.160284 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.160304 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.160312 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.160317 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.160322 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.160328 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.160333 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.160338 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.160343 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.160348 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.160353 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.160358 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.160364 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.160369 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.160375 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.160381 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.160386 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.160391 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.160397 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.160402 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.160407 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.160413 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.160418 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.160424 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.160430 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.160435 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.160441 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.160446 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.160456 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.160471 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.160477 | controller |       09:33:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.160482 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.160490 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.160496 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.160501 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.160507 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.160513 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.160518 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.160522 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.160527 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.160536 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.160541 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.160546 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.160551 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.160557 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.160562 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.160567 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.160578 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.160583 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.160588 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.160593 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.160598 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.160603 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.160609 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.160615 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.160620 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.160625 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.160630 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.160635 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.160640 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.160648 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.160653 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.160658 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.160663 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.160668 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.160673 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.160678 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.160683 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.160688 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:55 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.160693 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.160702 | controller |       09:33:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.160707 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.160713 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.160718 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.160723 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.160731 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.160736 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.160744 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.160750 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.160754 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.160760 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.160765 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.160770 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.160778 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.160790 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.160794 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.160801 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.160812 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.160817 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.160822 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.160827 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.160836 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.160842 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.160847 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.160852 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.160857 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.160862 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.160867 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.160872 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.160877 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.160882 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.160895 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.160900 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.160905 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.160910 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.160915 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.160920 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.160925 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.160930 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.160935 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.160940 | controller |       09:33:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.160959 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.160965 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.160970 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.160976 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.160982 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.160988 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.160994 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.161001 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.161007 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.161012 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.161017 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.161022 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.161028 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.161035 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.161040 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.161045 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.161050 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.161058 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.161063 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.161069 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.161074 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.161079 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.161084 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.161089 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.161094 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.161099 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.161105 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.161110 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.161115 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.161120 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.161125 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.161130 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.161135 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.161140 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.161145 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.161150 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.161155 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.161160 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.161165 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.161170 | controller |       09:33:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.161178 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.161183 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.161200 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.161206 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.161213 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.161218 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.161223 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.161228 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.161233 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.161238 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.161243 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.161248 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.161253 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.161258 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.161263 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.161270 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.161276 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.161281 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.161286 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.161308 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.161313 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.161318 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.161323 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.161328 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.161333 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.161338 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.161344 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.161349 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.161354 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.161359 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.161364 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.161369 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.161374 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.161380 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.161385 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.161390 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.161397 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.161405 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.161411 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.161416 | controller |       09:33:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.161421 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:33:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.161427 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.161432 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.161449 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.161456 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.161462 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.161468 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.161481 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.161491 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.161497 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.161502 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.161508 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.161514 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.161520 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.161529 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.161535 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.161541 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.161546 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.161552 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.161557 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.161563 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.161567 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.161571 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.161576 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.161580 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.161584 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.161589 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.161621 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.161627 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.161634 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.161638 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.161642 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.161646 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.161650 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.161655 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.161659 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.161663 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.161667 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:33:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.161671 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.161675 | controller |       09:33:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.161680 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.161684 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.161688 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.161692 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.161696 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.161711 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.161716 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.161720 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.161724 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.161729 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.161733 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.161737 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.161741 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.161745 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.161749 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.161753 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.161758 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.161762 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.161766 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.161770 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.161777 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.161782 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.161786 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.161790 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.161794 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.161798 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.161802 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.161806 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.161811 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.161815 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.161827 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.161831 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.161835 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.161839 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.161843 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.161848 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.161852 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.161856 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.161862 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.161866 | controller |       09:34:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.161870 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.161874 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.161878 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.161882 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.161887 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.161891 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.161906 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.161911 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.161916 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.161920 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.161924 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.161928 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.161932 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.161938 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.161943 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.161947 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.161951 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.161957 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.161963 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.161969 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.161976 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.161982 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.161988 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.161993 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.161999 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.162004 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.162009 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.162015 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.162020 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.162026 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.162031 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.162037 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.162042 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.162048 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.162054 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.162059 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.162065 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.162071 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:02 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.162076 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.162082 | controller |       09:34:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.162087 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.162093 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.162098 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.162103 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.162110 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.162114 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.162119 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.162132 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.162137 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.162141 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.162145 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.162150 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.162154 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.162161 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.162165 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.162169 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.162174 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.162178 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.162182 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.162186 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.162190 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.162195 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.162199 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.162204 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.162208 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.162212 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.162216 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.162220 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.162224 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.162228 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.162233 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.162237 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.162241 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.162245 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.162249 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.162253 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.162258 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.162263 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.162268 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.162272 | controller |       09:34:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.162276 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.162280 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.162284 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.162303 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.162312 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.162316 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.162320 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.162324 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.162329 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.162341 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.162345 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.162350 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.162354 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.162358 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.162363 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.162367 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.162371 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.162375 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.162379 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.162383 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.162388 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.162392 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.162396 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.162400 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.162404 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.162408 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.162412 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.162417 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.162421 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.162427 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.162431 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.162435 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.162439 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.162444 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.162448 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.162453 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.162458 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.162462 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:04 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.162466 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.162470 | controller |       09:34:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.162474 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.162484 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.162488 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.162492 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.162496 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.162500 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.162504 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.162509 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.162513 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.162517 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.162531 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.162536 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.162540 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.162546 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.162552 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.162568 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.162574 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.162580 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.162586 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.162592 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.162601 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.162607 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.162613 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.162619 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.162624 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.162630 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.162636 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.162642 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.162648 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.162654 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.162660 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.162665 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.162671 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.162676 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.162682 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.162688 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.162694 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.162700 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.162705 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.162711 | controller |       09:34:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.162717 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.162722 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.162728 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.162733 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.162739 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.162744 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.162750 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.162754 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.162758 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.162763 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.162767 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.162771 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.162790 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.162797 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.162801 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.162807 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.162812 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.162816 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.162820 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.162824 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.162829 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.162833 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.162837 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.162841 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.162845 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.162849 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.162854 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.162858 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.162862 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.162866 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.162870 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.162874 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.162878 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.162882 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.162887 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.162891 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.162895 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.162900 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.162905 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.162914 | controller |       09:34:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.162928 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.162934 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.162940 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.162945 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.162955 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.162964 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.162970 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.162976 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.162982 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.162988 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.162994 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.163000 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.163006 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.163022 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.163027 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.163031 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.163035 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.163040 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.163044 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.163048 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.163052 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.163060 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.163064 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.163068 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.163072 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.163077 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.163081 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.163085 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.163090 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.163096 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.163113 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.163119 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.163125 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.163134 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.163140 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.163146 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.163151 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.163158 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.163162 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.163166 | controller |       09:34:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.163171 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.163175 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.163179 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.163183 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.163187 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.163191 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.163195 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.163200 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.163204 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.163209 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.163214 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.163220 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.163225 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.163231 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.163236 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.163252 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.163259 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.163264 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.163270 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.163275 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.163281 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.163286 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.163312 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.163317 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.163323 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.163328 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.163333 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.163339 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.163345 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.163353 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.163361 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.163366 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.163372 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.163377 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.163383 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.163388 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.163394 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.163399 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:08 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.163405 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.163411 | controller |       09:34:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.163417 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.163422 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.163428 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.163432 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.163436 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.163440 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.163444 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.163448 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.163453 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.163457 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.163461 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.163468 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.163472 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.163476 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.163480 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.163485 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.163498 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.163502 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.163507 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.163511 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.163517 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.163521 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.163525 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.163529 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.163533 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.163538 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.163542 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.163546 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.163550 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.163554 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.163558 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.163563 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.163567 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.163571 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.163575 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.163579 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.163584 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.163588 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.163592 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.163596 | controller |       09:34:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.163600 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.163605 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.163609 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.163614 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.163620 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.163626 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.163631 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.163637 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.163643 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.163649 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.163655 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.163660 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.163665 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.163672 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.163676 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.163680 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.163684 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.163696 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.163700 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.163704 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.163709 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.163713 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.163717 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.163721 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.163725 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.163729 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.163734 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.163738 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.163744 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.163749 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.163753 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.163757 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.163761 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.163765 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.163769 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.163774 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.163778 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.163782 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:11 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.163786 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.163790 | controller |       09:34:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.163795 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.163799 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.163803 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.163807 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.163815 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.163820 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.163824 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.163828 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.163832 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.163836 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.163840 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.163845 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.163851 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.163856 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.163862 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.163868 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.163874 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.163880 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.163885 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.163899 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.163904 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.163909 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.163914 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.163919 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.163923 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.163928 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.163934 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.163941 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.163946 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.163952 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.163957 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.163962 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.163966 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.163970 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.163974 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.163978 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.163982 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.163989 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:12 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.163993 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.163997 | controller |       09:34:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.164002 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.164006 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.164010 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.164014 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.164018 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.164023 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.164027 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.164031 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.164035 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.164039 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.164046 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.164050 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.164054 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.164058 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.164063 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.164067 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.164071 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.164075 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.164079 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.164083 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.164098 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.164103 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.164107 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.164112 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.164116 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.164120 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.164124 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.164128 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.164133 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.164138 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.164143 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.164147 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.164151 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.164155 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.164159 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.164164 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.164168 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.164172 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:13 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.164176 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.164180 | controller |       09:34:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.164184 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.164189 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.164193 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.164197 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.164201 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.164205 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.164210 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.164214 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.164218 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.164222 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.164226 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.164231 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.164235 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.164239 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.164243 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.164249 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.164253 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.164257 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.164261 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.164266 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.164273 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.164283 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.164290 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.164309 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.164313 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.164317 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.164322 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.164328 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.164332 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.164337 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.164341 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.164345 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.164349 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.164353 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.164357 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.164362 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.164366 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.164370 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:14 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.164374 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.164378 | controller |       09:34:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.164383 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.164387 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.164391 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.164396 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.164400 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.164404 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.164408 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.164412 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.164416 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.164421 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.164425 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.164429 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.164433 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.164439 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.164443 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.164447 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.164451 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.164455 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.164460 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.164464 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.164468 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.164472 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.164483 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.164488 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.164492 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.164496 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.164500 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.164504 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.164509 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.164513 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.164517 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.164521 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.164525 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.164529 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.164534 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.164538 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.164542 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.164546 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:15 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.164550 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.164554 | controller |       09:34:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.164559 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.164565 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.164570 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.164574 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.164580 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.164584 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.164589 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.164593 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.164597 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.164602 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.164607 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.164611 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.164616 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.164620 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.164624 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.164629 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.164633 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.164637 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.164641 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.164645 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.164649 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.164654 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.164658 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.164662 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.164672 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.164677 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.164681 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.164685 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.164689 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.164693 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.164700 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.164704 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.164708 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.164713 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.164717 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.164721 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.164725 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.164731 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:16 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.164735 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.164739 | controller |       09:34:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.164744 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.164748 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.164752 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.164756 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.164760 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.164764 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.164769 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.164773 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.164778 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.164782 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.164787 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.164791 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.164795 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.164799 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.164803 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.164807 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.164811 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.164816 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.164820 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.164824 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.164828 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.164832 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.164836 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.164841 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.164845 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.164850 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.164860 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.164864 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.164868 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.164875 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.164879 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.164883 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.164887 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.164891 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.164896 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.164900 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.164904 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.164908 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:18 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.164912 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.164916 | controller |       09:34:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.164921 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.164925 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.164929 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.164933 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.164937 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.164941 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.164946 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.164950 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.164954 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.164958 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.164962 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.164966 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.164970 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.164975 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.164979 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.164983 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.164987 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.164991 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.164996 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.165000 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.165007 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.165011 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.165016 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.165020 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.165024 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.165028 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.165033 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.165042 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.165047 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.165051 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.165055 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.165060 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.165064 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.165068 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.165072 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.165077 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.165081 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.165085 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:19 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.165089 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.165094 | controller |       09:34:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.165098 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.165102 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.165106 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.165111 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.165119 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.165123 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.165128 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.165132 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.165137 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.165142 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.165146 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.165150 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.165154 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.165160 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.165164 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.165169 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.165173 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.165177 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.165182 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.165186 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.165190 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.165194 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.165199 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.165203 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.165207 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.165211 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.165216 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.165220 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.165229 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.165234 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.165238 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.165242 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.165247 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.165251 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.165255 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.165260 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.165265 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.165270 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:20 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.165304 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.165319 | controller |       09:34:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.165325 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.165331 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.165336 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.165342 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.165351 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.165357 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.165362 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.165366 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.165371 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.165375 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.165379 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.165383 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.165387 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.165391 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.165395 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.165400 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.165404 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.165408 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.165412 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.165416 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.165420 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.165425 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.165429 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.165433 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.165440 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.165444 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.165448 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.165452 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.165456 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.165460 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.165474 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.165479 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.165483 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.165487 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.165491 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.165496 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.165500 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.165505 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:21 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.165510 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.165514 | controller |       09:34:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.165518 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.165522 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.165526 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.165531 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.165535 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.165539 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.165543 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.165547 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.165551 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.165556 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.165560 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.165564 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.165568 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.165572 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.165576 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.165583 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.165587 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.165591 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.165596 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.165600 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.165623 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.165629 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.165634 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.165640 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.165644 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.165649 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.165654 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.165659 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.165664 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.165671 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.165676 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.165688 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.165693 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.165698 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.165703 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.165708 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.165713 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.165717 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:22 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.165722 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.165727 | controller |       09:34:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.165732 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.165737 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.165742 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.165749 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.165754 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.165759 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.165764 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.165769 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.165774 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.165778 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.165783 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.165788 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.165793 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.165798 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.165803 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.165807 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.165812 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.165817 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.165822 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.165827 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.165833 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.165838 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.165843 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.165848 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.165853 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.165858 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.165862 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.165867 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.165871 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.165875 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.165879 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.165883 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.165893 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.165898 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.165902 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.165906 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.165911 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.165917 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:23 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.165921 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.165926 | controller |       09:34:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.165931 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.165936 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.165940 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.165944 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.165948 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.165952 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.165957 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.165961 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.165965 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.165969 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.165973 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.165977 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.165982 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.165987 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.165991 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.165995 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.166000 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.166004 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.166008 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.166012 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.166018 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.166022 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.166026 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.166032 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.166036 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.166040 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.166044 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.166048 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.166052 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.166057 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.166062 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.166066 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.166071 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.166075 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.166084 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.166089 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.166093 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.166097 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:24 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.166102 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.166109 | controller |       09:34:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.166113 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.166117 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.166122 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.166126 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.166132 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.166136 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.166140 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.166144 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.166148 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.166152 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.166156 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.166161 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.166165 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.166169 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.166173 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.166177 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.166181 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.166186 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.166190 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.166194 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.166198 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.166202 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.166206 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.166211 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.166215 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.166219 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.166223 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.166227 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.166231 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.166236 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.166240 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.166244 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.166248 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.166252 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.166257 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.166266 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.166270 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.166277 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:25 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.166283 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.166321 | controller |       09:34:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.166330 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.166336 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.166346 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.166352 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.166358 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.166364 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.166369 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.166375 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.166381 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.166387 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.166391 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.166395 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.166401 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.166406 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.166412 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.166418 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.166424 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.166429 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.166435 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.166441 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.166447 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.166452 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.166458 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.166464 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.166469 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.166475 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.166481 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.166487 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.166493 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.166505 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.166509 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.166513 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.166517 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.166522 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.166526 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.166530 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.166543 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.166547 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:27 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.166552 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.166556 | controller |       09:34:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.166560 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.166564 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.166568 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.166573 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.166579 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.166584 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.166588 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.166592 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.166596 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.166600 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.166604 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.166608 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.166613 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.166617 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.166621 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.166625 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.166629 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.166634 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.166638 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.166642 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.166648 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.166654 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.166659 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.166663 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.166667 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.166671 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.166676 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.166680 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.166684 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.166688 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.166692 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.166697 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.166701 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.166705 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.166709 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.166714 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.166718 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.166722 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:28 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.166732 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.166737 | controller |       09:34:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.166741 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.166745 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.166750 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.166754 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.166758 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.166762 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.166767 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.166771 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.166775 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.166779 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.166783 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.166788 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.166792 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.166798 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.166802 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.166806 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.166810 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.166815 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.166819 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.166823 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.166827 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.166832 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.166836 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.166840 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.166844 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.166848 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.166853 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.166857 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.166861 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.166865 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.166869 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.166874 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.166878 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.166882 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.166886 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.166891 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.166895 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.166899 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:29 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.166903 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.166913 | controller |       09:34:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.166917 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.166923 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.166928 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.166932 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.166937 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.166942 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.166946 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.166950 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.166954 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.166958 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.166962 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.166967 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.166971 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.166975 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.166979 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.166985 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.166989 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.166993 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.166997 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.167001 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.167007 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.167011 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.167015 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.167019 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.167023 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.167027 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.167032 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.167036 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.167040 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.167044 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.167048 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.167052 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.167056 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.167061 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.167065 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.167069 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.167073 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.167221 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:30 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.167226 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.167231 | controller |       09:34:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.167247 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.167262 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.167268 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.167274 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.167280 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.167286 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.167313 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.167319 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.167325 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.167330 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.167336 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.167341 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.167346 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.167352 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.167357 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.167362 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.167371 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.167376 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.167382 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.167388 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.167393 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.167399 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.167403 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.167407 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.167411 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.167415 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.167419 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.167423 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.167428 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.167436 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.167440 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.167444 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.167448 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.167452 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.167457 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.167461 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.167465 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.167469 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:31 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.167473 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.167477 | controller |       09:34:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.167482 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.167486 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.167498 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.167503 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.167508 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.167512 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.167516 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.167520 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.167524 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.167528 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.167533 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.167537 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.167541 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.167545 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.167549 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.167553 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.167557 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.167561 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.167566 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.167570 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.167576 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.167580 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.167584 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.167588 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.167592 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.167596 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.167600 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.167605 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.167609 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.167613 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.167619 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.167624 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.167628 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.167632 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.167636 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.167640 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.167644 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.167648 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:32 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.167653 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.167657 | controller |       09:34:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.167662 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.167667 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.167671 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.167680 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.167685 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.167691 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.167695 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.167699 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.167703 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.167707 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.167712 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.167717 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.167722 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.167737 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.167749 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.167756 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.167762 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.167767 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.167773 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.167779 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.167785 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.167790 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.167796 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.167802 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.167808 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.167812 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.167816 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.167821 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.167825 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.167829 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.167833 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.167837 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.167843 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.167848 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.167854 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.167860 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.167866 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.167872 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:34 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.167878 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.167884 | controller |       09:34:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.167889 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.167895 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.167900 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.167904 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.167911 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.167924 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.167931 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.167937 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.167943 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.167949 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.167955 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.167960 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.167966 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.167972 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.167981 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.167987 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.167993 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.167999 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.168005 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.168011 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.168016 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.168022 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.168028 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.168034 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.168039 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.168046 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.168052 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.168058 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.168064 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.168070 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.168076 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.168082 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.168088 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.168094 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.168100 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.168106 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.168112 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.168119 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:35 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.168123 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.168127 | controller |       09:34:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.168132 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.168136 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.168141 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.168147 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.168156 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.168162 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.168178 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.168185 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.168191 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.168197 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.168203 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.168209 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.168215 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.168220 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.168226 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.168230 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.168234 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.168239 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.168245 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.168251 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.168260 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.168267 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.168272 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.168278 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.168284 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.168309 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.168315 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.168321 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.168327 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.168335 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.168341 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.168347 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.168353 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.168359 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.168364 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.168370 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.168376 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.168382 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:36 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.168390 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.168396 | controller |       09:34:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.168402 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.168407 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.168413 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.168419 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.168425 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.168430 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.168436 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.168450 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.168458 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.168463 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.168469 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.168475 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.168482 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.168488 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.168494 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.168499 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.168506 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.168511 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.168518 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.168524 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.168533 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.168540 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.168546 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.168553 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.168559 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.168565 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.168571 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.168576 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.168582 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.168588 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.168594 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.168599 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.168605 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.168611 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.168617 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.168623 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.168629 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.168635 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:37 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.168641 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.168647 | controller |       09:34:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.168653 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.168659 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.168665 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.168671 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.168676 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.168682 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.168688 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.168694 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.168700 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.168715 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.168722 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.168728 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.168734 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.168748 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.168754 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.168761 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.168767 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.168773 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.168779 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.168785 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.168790 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.168796 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.168802 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.168808 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.168814 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.168820 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.168825 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.168831 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.168837 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.168843 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.168849 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.168854 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.168860 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.168866 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.168872 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.168877 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.168883 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.168889 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:38 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.168895 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.168901 | controller |       09:34:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.168907 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.168913 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.168920 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.168926 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.168935 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.168943 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.168948 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.168954 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.168959 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.168964 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.168979 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.168986 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.168992 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.168997 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.169003 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.169009 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.169015 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.169020 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.169026 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.169032 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.169038 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.169044 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.169049 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.169055 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.169107 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.169138 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.169144 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.169149 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.169155 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.169161 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.169166 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.169174 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.169179 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.169183 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.169187 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.169193 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.169197 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.169203 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:39 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.169207 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.169211 | controller |       09:34:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.169216 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.169220 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.169224 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.169228 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.169232 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.169236 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.169241 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.169245 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.169249 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.169253 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.169257 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.169261 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.169274 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.169279 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.169283 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.169289 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.169311 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.169316 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.169320 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.169324 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.169328 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.169332 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.169337 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.169341 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.169345 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.169349 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.169353 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.169357 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.169361 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.169367 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.169386 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.169391 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.169395 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.169399 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.169403 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.169407 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.169412 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.169416 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:41 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.169420 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.169424 | controller |       09:34:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.169428 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.169432 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.169437 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.169441 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.169445 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.169449 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.169453 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.169458 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.169462 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.169466 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.169470 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.169474 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.169480 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.169490 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.169495 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.169499 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.169503 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.169507 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.169511 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.169516 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.169523 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.169527 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.169531 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.169535 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.169540 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.169544 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.169548 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.169552 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.169556 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.169560 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.169565 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.169569 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.169573 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.169577 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.169581 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.169586 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.169590 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.169594 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:42 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.169598 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.169602 | controller |       09:34:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.169607 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.169630 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.169636 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.169640 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.169644 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.169649 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.169653 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.169657 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.169661 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.169665 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.169669 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.169674 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.169678 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.169684 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.169688 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.169699 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.169703 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.169708 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.169712 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.169716 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.169720 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.169724 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.169729 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.169733 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.169737 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.169741 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.169745 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.169749 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.169754 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.169758 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.169762 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.169766 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.169774 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.169779 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.169785 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.169791 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.169797 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.169803 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:43 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.169808 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.169815 | controller |       09:34:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.169820 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.169825 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.169829 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.169833 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.169842 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.169846 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.169850 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.169854 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.169858 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.169863 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.169867 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.169871 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.169875 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.169879 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.169884 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.169888 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.169899 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.169903 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.169908 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.169912 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.169916 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.169920 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.169924 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.169929 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.169933 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.169937 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.169941 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.169945 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.169949 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.169954 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.169958 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.169962 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.169966 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.169970 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.169975 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.169979 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.169983 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.169990 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:44 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.169995 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.169999 | controller |       09:34:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.170003 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.170007 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.170011 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.170016 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.170022 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.170028 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.170034 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.170040 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.170046 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.170052 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.170057 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.170066 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.170072 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.170078 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.170082 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.170087 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.170091 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.170102 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.170107 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.170111 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.170116 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.170120 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.170124 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.170128 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.170132 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.170137 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.170141 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.170145 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.170149 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.170155 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.170159 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.170163 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.170168 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.170172 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.170176 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.170180 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.170184 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.170189 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:45 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.170193 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.170197 | controller |       09:34:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.170201 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.170205 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.170209 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.170214 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.170218 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.170222 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.170226 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.170230 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.170235 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.170239 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.170243 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.170247 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.170251 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.170255 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.170259 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.170265 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.170270 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.170274 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.170283 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.170287 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.170309 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.170314 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.170318 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.170322 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.170327 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.170331 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.170335 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.170339 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.170345 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.170350 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.170354 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.170358 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.170362 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.170366 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.170371 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.170375 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.170379 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.170383 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:46 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.170389 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.170393 | controller |       09:34:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.170397 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.170402 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.170406 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.170410 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.170414 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.170418 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.170422 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.170427 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.170431 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.170435 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.170439 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.170443 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.170447 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.170453 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.170457 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.170461 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.170466 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.170470 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.170474 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.170478 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.170490 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.170494 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.170499 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.170503 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.170507 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.170511 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.170516 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.170522 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.170528 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.170533 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.170539 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.170545 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.170550 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.170556 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.170561 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.170567 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.170573 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.170579 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:47 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.170585 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.170591 | controller |       09:34:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.170596 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.170602 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.170608 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.170614 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.170623 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.170630 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.170636 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.170641 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.170647 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.170653 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.170662 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.170668 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.170674 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.170680 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.170686 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.170691 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.170697 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.170702 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.170708 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.170713 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.170719 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.170732 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.170737 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.170742 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.170748 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.170764 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.170771 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.170777 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.170783 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.170788 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.170798 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.170804 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.170810 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.170815 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.170821 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.170827 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.170833 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.170841 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:49 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.170845 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.170851 | controller |       09:34:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.170864 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.170875 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.170881 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.170887 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.170893 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.170899 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.170905 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.170911 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.170917 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.170923 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.170930 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.170936 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.170942 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.170949 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.170956 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.170963 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.170969 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.170977 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.170983 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.170989 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.170994 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.171000 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.171016 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.171022 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.171028 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.171034 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.171038 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.171046 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.171050 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.171057 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.171061 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.171065 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.171070 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.171074 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.171078 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.171082 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.171086 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.171090 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:50 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.171095 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.171099 | controller |       09:34:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.171103 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.171107 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.171111 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.171115 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.171119 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.171124 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.171128 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.171132 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.171136 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.171140 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.171144 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.171148 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.171152 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.171157 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.171161 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.171165 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.171169 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.171173 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.171177 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.171181 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.171372 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.171379 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.171384 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.171388 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.171398 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.171402 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.171406 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.171410 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.171415 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.171419 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.171423 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.171427 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.171431 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.171435 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.171440 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.171444 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.171448 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.171452 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:51 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.171456 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.171460 | controller |       09:34:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.171464 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.171469 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.171473 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.171477 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.171484 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.171488 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.171492 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.171496 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.171501 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.171506 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.171510 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.171515 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.171519 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.171524 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.171528 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.171533 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.171537 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.171541 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.171545 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.171549 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.171553 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.171558 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.171562 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.171566 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.171570 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.171580 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.171584 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.171588 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.171592 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.171596 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.171601 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.171605 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.171609 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.171613 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.171617 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.171621 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.171626 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.171630 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:52 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.171634 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.171638 | controller |       09:34:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.171642 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.171646 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.171651 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.171655 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.171662 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.171666 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.171671 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.171675 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.171679 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.171683 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.171687 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.171691 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.171695 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.171700 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.171704 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.171708 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.171712 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.171716 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.171720 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.171724 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.171730 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.171734 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.171738 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.171743 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.171747 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.171752 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.171756 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.171766 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.171770 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.171774 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.171779 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.171784 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.171788 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.171792 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.171796 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.171800 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.171805 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.171815 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:53 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.171819 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.171823 | controller |       09:34:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.171828 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.171832 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.171836 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.171840 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.171844 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.171848 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.171853 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.171857 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.171861 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.171865 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.171869 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.171873 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.171877 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.171882 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.171886 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.171892 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.171896 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.171900 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.171904 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.171909 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.171913 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.171917 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.171921 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.171927 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.171931 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.171935 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.171939 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.171943 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.171953 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.171959 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.171963 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.171967 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.171971 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.171976 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.171980 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.171984 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.171988 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.171992 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:54 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.171996 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.172001 | controller |       09:34:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.172005 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.172009 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.172013 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.172017 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.172021 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.172026 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.172030 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.172034 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.172040 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.172044 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.172048 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.172052 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.172056 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.172060 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.172065 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.172069 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.172073 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.172077 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.172081 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.172085 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.172091 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.172095 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.172100 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.172104 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.172108 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.172112 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.172116 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.172120 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.172124 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.172128 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.172137 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.172142 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.172146 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.172150 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.172155 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.172159 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.172163 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.172167 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:56 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.172171 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.172175 | controller |       09:34:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.172180 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.172184 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.172188 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.172192 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.172196 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.172200 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.172205 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.172209 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.172214 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.172219 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.172225 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.172239 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.172247 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.172257 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.172263 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.172268 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.172274 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.172279 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.172285 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.172313 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.172318 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.172322 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.172327 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.172331 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.172338 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.172342 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.172347 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.172351 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.172355 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.172359 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.172366 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.172378 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.172383 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.172388 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.172392 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.172396 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.172401 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.172405 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:57 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.172409 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.172414 | controller |       09:34:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.172418 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.172422 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.172426 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.172431 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.172437 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.172441 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.172445 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.172449 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.172454 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.172458 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.172462 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.172466 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.172470 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.172475 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.172479 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.172483 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.172487 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.172491 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.172496 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.172500 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.172504 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.172508 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.172513 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.172517 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.172521 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.172525 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.172529 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.172533 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.172538 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.172542 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.172546 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.172550 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.172560 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.172565 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.172569 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.172573 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.172578 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.172583 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:58 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.172588 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.172592 | controller |       09:34:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.172596 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:34:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.172600 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.172605 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.172611 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.172617 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.172633 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.172641 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.172647 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.172653 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.172660 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.172666 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.172671 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.172677 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.172682 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.172686 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.172690 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.172694 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.172698 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.172702 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.172707 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.172714 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.172718 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.172722 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.172726 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.172731 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.172735 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.172739 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.172743 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.172747 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.172754 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.172758 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.172762 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.172766 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.172779 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.172784 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.172788 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.172792 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.172796 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:34:59 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.172800 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.172805 | controller |       09:34:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.172809 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:35:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.172813 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.172817 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.172821 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.172827 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.172832 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.172837 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.172841 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.172845 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.172849 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.172853 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.172857 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.172862 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.172866 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.172870 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.172874 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.172878 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.172882 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.172887 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.172891 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.172896 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.172901 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.172905 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.172909 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.172913 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.172917 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.172921 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.172927 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.172931 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.172935 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.172939 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.172944 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.172948 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.172952 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.172956 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.172965 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.172970 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.172974 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:35:00 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.172978 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.172982 | controller |       09:35:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.172986 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:35:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.172991 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.172995 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.172999 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.173003 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.173007 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.173011 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.173015 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.173019 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.173024 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.173028 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.173032 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.173036 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.173041 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.173046 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.173050 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.173054 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.173058 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.173062 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.173066 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.173070 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.173075 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.173079 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.173083 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.173087 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.173091 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.173095 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.173099 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.173103 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.173108 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.173112 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.173116 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.173120 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.173124 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.173128 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.173133 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.173141 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.173146 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:35:01 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.173150 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.173154 | controller |       09:35:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.173159 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:35:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.173163 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.173168 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.173173 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.173178 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.173182 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.173187 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.173191 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.173195 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.173199 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.173203 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.173207 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.173211 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.173216 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.173220 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.173225 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.173229 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.173234 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.173238 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.173242 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.173246 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.173250 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.173254 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.173258 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.173263 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.173267 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.173271 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.173275 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.173279 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.173284 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.173304 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.173312 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.173316 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.173320 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.173325 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.173329 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.173333 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.173340 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:35:03 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.173349 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.173354 | controller |       09:35:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.173358 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:35:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.173363 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.173367 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.173371 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.173375 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.173380 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.173384 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.173388 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.173392 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.173397 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.173401 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.173405 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.173409 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.173413 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.173418 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.173422 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.173426 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.173430 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.173434 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.173439 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.173443 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.173451 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.173456 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.173460 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.173464 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.173468 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.173473 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.173477 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.173481 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.173487 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.173491 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.173495 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.173499 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.173504 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.173508 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.173513 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.173517 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.173522 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:35:05 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.173526 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.173535 | controller |       09:35:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.173539 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:35:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.173544 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.173548 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.173552 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.173556 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.173561 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.173565 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.173569 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.173573 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.173577 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.173582 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.173586 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.173590 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.173594 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.173599 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.173603 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.173607 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.173611 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.173615 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.173640 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.173649 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.173653 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.173657 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.173661 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.173665 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.173669 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.173674 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.173678 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.173682 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.173686 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.173693 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.173697 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.173701 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.173705 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.173709 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.173714 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.173718 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.173722 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:35:06 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.173726 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.173730 | controller |       09:35:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.173739 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:35:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.173746 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.173750 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.173754 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.173758 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.173763 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.173767 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.173771 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.173775 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.173779 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.173783 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.173787 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.173792 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.173797 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.173801 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.173806 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.173810 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.173814 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.173818 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.173822 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.173826 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.173830 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.173835 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.173839 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.173843 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.173847 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.173851 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.173855 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.173859 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.173864 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.173868 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.173872 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.173876 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.173880 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.173884 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.173889 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.173893 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.173897 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:35:07 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.173901 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.173905 | controller |       09:35:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.173910 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:35:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.173916 | controller |       | running command: [sh -c set -euo pipefail\n        NS=\"${NAMESPACE}\"\n        \n
2026-03-14 09:44:27.173937 | controller |       \       echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n
2026-03-14 09:44:27.173948 | controller |       \       oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n
2026-03-14 09:44:27.173958 | controller |       \       \n        ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n
2026-03-14 09:44:27.173964 | controller |       \       if [ -z \"${ac_id}\" ]; then\n          echo \"ERROR: appcred/ac-watcher.status.acID
2026-03-14 09:44:27.173970 | controller |       is empty\"\n          exit 1\n        fi\n        echo \"ac-watcher.status.acID=${ac_id}\"\n
2026-03-14 09:44:27.173976 | controller |       \       \n        oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n        secret_ac_id=$(oc
2026-03-14 09:44:27.173981 | controller |       get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n
2026-03-14 09:44:27.173985 | controller |       \       if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n          echo \"ERROR:
2026-03-14 09:44:27.173990 | controller |       Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n          exit
2026-03-14 09:44:27.173994 | controller |       1\n        fi\n        echo \"✓ Secret AC_ID matches appcred.status.acID\"\n        \n
2026-03-14 09:44:27.173998 | controller |       \       echo \"Waiting for watcher rollouts...\"\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.174002 | controller |       statefulset/watcher-kuttl-api --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.174006 | controller |       statefulset/watcher-kuttl-applier --timeout=300s\n        oc rollout status -n \"${NS}\"
2026-03-14 09:44:27.174010 | controller |       statefulset/watcher-kuttl-decision-engine --timeout=300s\n        \n        old_api_uid=$(oc
2026-03-14 09:44:27.174017 | controller |       get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n
2026-03-14 09:44:27.174022 | controller |       \       old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n
2026-03-14 09:44:27.174026 | controller |       \       old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n
2026-03-14 09:44:27.174030 | controller |       \       \n        new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.174034 | controller |       \       new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n
2026-03-14 09:44:27.174038 | controller |       \       new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o
2026-03-14 09:44:27.174043 | controller |       jsonpath='{.metadata.uid}')\n        \n        if [ \"${old_api_uid}\" = \"${new_api_uid}\"
2026-03-14 09:44:27.174047 | controller |       ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\"
2026-03-14 09:44:27.174052 | controller |       = \"${new_de_uid}\" ]; then\n          echo \"ERROR: expected watcher pods to restart
2026-03-14 09:44:27.174058 | controller |       after appcred secret became available\"\n          echo \"  api: ${old_api_uid}
2026-03-14 09:44:27.174073 | controller |       -> ${new_api_uid}\"\n          echo \"  applier: ${old_applier_uid} -> ${new_applier_uid}\"\n
2026-03-14 09:44:27.174081 | controller |       \         echo \"  decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n          exit
2026-03-14 09:44:27.174086 | controller |       1\n        fi\n        echo \"✓ watcher pods restarted after appcred secret became
2026-03-14 09:44:27.174092 | controller |       available\"\n        \n        echo \"Checking watcher config contains application_credential_id...\"\n
2026-03-14 09:44:27.174097 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n          bash
2026-03-14 09:44:27.174103 | controller |       -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.174109 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n
2026-03-14 09:44:27.174114 | controller |       \         bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n
2026-03-14 09:44:27.174119 | controller |       \       oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.174125 | controller |       -- \\\n          bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\"
2026-03-14 09:44:27.174130 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\"\n        echo \"✓ watcher config contains
2026-03-14 09:44:27.174139 | controller |       expected application_credential_id\"]\n    logger.go:42: 09:35:09 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.174144 | controller |       | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n    logger.go:42:
2026-03-14 09:44:27.174149 | controller |       09:35:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org
2026-03-14 09:44:27.174155 | controller |       \"ac-watcher\" not found\n    logger.go:42: 09:35:10 | appcred-tests/2-deploy-appcred
2026-03-14 09:44:27.174160 | controller |       | test step failed 2-deploy-appcred\n    case.go:396: failed in step 2-deploy-appcred\n
2026-03-14 09:44:27.174165 | controller |       \   case.go:398: command \"NS=\\\"${NAMESPACE}\\\"\\\\n echo \\\"Waiting for KeystoneApplicationCredential...\"
2026-03-14 09:44:27.174181 | controller |       failed, exit status 1\n    logger.go:42: 09:35:10 | appcred-tests | skipping kubernetes
2026-03-14 09:44:27.174187 | controller |       event logging\n=== CONT  kuttl/harness/watcher-notification\n    logger.go:42: 09:35:10
2026-03-14 09:44:27.174194 | controller |       | watcher-notification | Skipping creation of user-supplied namespace: watcher-kuttl-default\n
2026-03-14 09:44:27.174199 | controller |       \   logger.go:42: 09:35:10 | watcher-notification/0-cleanup-watcher | starting test
2026-03-14 09:44:27.174203 | controller |       step 0-cleanup-watcher\n    logger.go:42: 09:35:10 | watcher-notification/0-cleanup-watcher
2026-03-14 09:44:27.174207 | controller |       | test step completed 0-cleanup-watcher\n    logger.go:42: 09:35:10 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174212 | controller |       | starting test step 1-deploy-with-notification\n    logger.go:42: 09:35:10 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174216 | controller |       | Watcher:watcher-kuttl-default/watcher-kuttl created\n    logger.go:42: 09:35:10
2026-03-14 09:44:27.174220 | controller |       | watcher-notification/1-deploy-with-notification | running command: [sh -c set
2026-03-14 09:44:27.174224 | controller |       -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc get
2026-03-14 09:44:27.174229 | controller |       pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:44:27.174233 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.174237 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep
2026-03-14 09:44:27.174241 | controller |       -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.174245 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.174249 | controller |       09:35:10 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174253 | controller |       \   logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification | Now
2026-03-14 09:44:27.174257 | controller |       using project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.174261 | controller |       \   logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174266 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.174270 | controller |       \   logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174274 | controller |       head -1\n    logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174278 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174282 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174286 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174310 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:44:27.174315 | controller |       \   logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:44:27.174319 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.174323 | controller |       \   logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification | error:
2026-03-14 09:44:27.174327 | controller |       Internal error occurred: error executing command in container: container is not
2026-03-14 09:44:27.174332 | controller |       created or running\n    logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174336 | controller |       | ++ echo\n    logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174340 | controller |       | + '[' 0 == 1 ']'\n    logger.go:42: 09:35:12 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174344 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174350 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.174354 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.174358 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.174362 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.174366 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.174370 | controller |       09:35:12 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174375 | controller |       \   logger.go:42: 09:35:13 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.174379 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.174389 | controller |       \   logger.go:42: 09:35:13 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174394 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.174398 | controller |       \   logger.go:42: 09:35:13 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174402 | controller |       head -1\n    logger.go:42: 09:35:13 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174406 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:13 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174410 | controller |       | + APIPOD=\n    logger.go:42: 09:35:14 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174414 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174418 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.174423 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.174427 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.174433 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.174437 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.174441 | controller |       09:35:14 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174445 | controller |       \   logger.go:42: 09:35:14 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.174449 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.174453 | controller |       \   logger.go:42: 09:35:14 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174459 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.174463 | controller |       \   logger.go:42: 09:35:14 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174467 | controller |       head -1\n    logger.go:42: 09:35:14 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174471 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:14 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174476 | controller |       | + APIPOD=\n    logger.go:42: 09:35:15 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174480 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174484 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.174488 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.174492 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.174496 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.174500 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.174504 | controller |       09:35:15 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174509 | controller |       \   logger.go:42: 09:35:15 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.174513 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.174517 | controller |       \   logger.go:42: 09:35:15 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174521 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.174525 | controller |       \   logger.go:42: 09:35:15 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174529 | controller |       head -1\n    logger.go:42: 09:35:15 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174533 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:15 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174537 | controller |       | + APIPOD=\n    logger.go:42: 09:35:16 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174542 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174546 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.174550 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.174554 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.174558 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.174562 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.174575 | controller |       09:35:16 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174580 | controller |       \   logger.go:42: 09:35:17 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.174584 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.174588 | controller |       \   logger.go:42: 09:35:17 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174592 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.174596 | controller |       \   logger.go:42: 09:35:17 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174601 | controller |       head -1\n    logger.go:42: 09:35:17 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174607 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:17 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174612 | controller |       | + APIPOD=\n    logger.go:42: 09:35:18 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174616 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174620 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.174624 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.174628 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.174632 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.174637 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.174641 | controller |       09:35:18 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174645 | controller |       \   logger.go:42: 09:35:18 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.174649 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.174653 | controller |       \   logger.go:42: 09:35:18 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174657 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.174661 | controller |       \   logger.go:42: 09:35:18 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174666 | controller |       head -1\n    logger.go:42: 09:35:18 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174670 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:18 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174674 | controller |       | + APIPOD=\n    logger.go:42: 09:35:19 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174678 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174682 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.174686 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.174690 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.174695 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.174699 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.174703 | controller |       09:35:19 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174707 | controller |       \   logger.go:42: 09:35:19 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.174711 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.174715 | controller |       \   logger.go:42: 09:35:19 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174719 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.174724 | controller |       \   logger.go:42: 09:35:19 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174728 | controller |       head -1\n    logger.go:42: 09:35:19 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174732 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:19 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174736 | controller |       | + APIPOD=\n    logger.go:42: 09:35:20 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174740 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174746 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.174750 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.174754 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.174764 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.174769 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.174773 | controller |       09:35:20 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174777 | controller |       \   logger.go:42: 09:35:20 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.174781 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.174786 | controller |       \   logger.go:42: 09:35:20 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174790 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.174794 | controller |       \   logger.go:42: 09:35:20 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174799 | controller |       head -1\n    logger.go:42: 09:35:20 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174805 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:21 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174811 | controller |       | + APIPOD=\n    logger.go:42: 09:35:22 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174819 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174825 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.174831 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.174837 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.174842 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.174848 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.174856 | controller |       09:35:22 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174862 | controller |       \   logger.go:42: 09:35:22 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.174867 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.174873 | controller |       \   logger.go:42: 09:35:22 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174879 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.174884 | controller |       \   logger.go:42: 09:35:22 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174890 | controller |       head -1\n    logger.go:42: 09:35:22 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174896 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:22 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174901 | controller |       | + APIPOD=\n    logger.go:42: 09:35:23 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174907 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174912 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.174917 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.174922 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.174930 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.174936 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.174942 | controller |       09:35:23 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174948 | controller |       \   logger.go:42: 09:35:23 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.174954 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.174960 | controller |       \   logger.go:42: 09:35:23 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174966 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.174971 | controller |       \   logger.go:42: 09:35:23 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.174977 | controller |       head -1\n    logger.go:42: 09:35:23 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174983 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:23 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174988 | controller |       | + APIPOD=\n    logger.go:42: 09:35:24 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.174994 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.174999 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.175014 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.175020 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.175027 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.175033 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.175042 | controller |       09:35:24 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.175048 | controller |       \   logger.go:42: 09:35:24 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.175054 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.175060 | controller |       \   logger.go:42: 09:35:24 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.175066 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.175071 | controller |       \   logger.go:42: 09:35:24 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.175077 | controller |       head -1\n    logger.go:42: 09:35:24 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175081 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:24 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175085 | controller |       | + APIPOD=\n    logger.go:42: 09:35:25 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175090 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.175094 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.175098 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.175102 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.175106 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.175112 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.175118 | controller |       09:35:25 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.175130 | controller |       \   logger.go:42: 09:35:26 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.175136 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.175142 | controller |       \   logger.go:42: 09:35:26 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.175148 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.175154 | controller |       \   logger.go:42: 09:35:26 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.175160 | controller |       head -1\n    logger.go:42: 09:35:26 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175166 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:26 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175171 | controller |       | + APIPOD=\n    logger.go:42: 09:35:27 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175177 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.175183 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.175188 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.175192 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.175197 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.175203 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.175210 | controller |       09:35:27 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.175215 | controller |       \   logger.go:42: 09:35:27 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.175221 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.175227 | controller |       \   logger.go:42: 09:35:27 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.175233 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.175239 | controller |       \   logger.go:42: 09:35:27 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.175245 | controller |       head -1\n    logger.go:42: 09:35:27 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175252 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:27 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175258 | controller |       | + APIPOD=\n    logger.go:42: 09:35:28 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175272 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.175279 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.175285 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.175311 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.175317 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.175323 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.175328 | controller |       09:35:28 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.175333 | controller |       \   logger.go:42: 09:35:28 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.175339 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.175348 | controller |       \   logger.go:42: 09:35:28 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.175353 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.175359 | controller |       \   logger.go:42: 09:35:28 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.175364 | controller |       head -1\n    logger.go:42: 09:35:28 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175370 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:28 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175376 | controller |       | + APIPOD=\n    logger.go:42: 09:35:29 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175381 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.175387 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.175392 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.175398 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.175403 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.175409 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.175414 | controller |       09:35:29 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.175420 | controller |       \   logger.go:42: 09:35:30 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.175426 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.175432 | controller |       \   logger.go:42: 09:35:30 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.175437 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.175443 | controller |       \   logger.go:42: 09:35:30 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.175452 | controller |       head -1\n    logger.go:42: 09:35:30 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175458 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:30 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175464 | controller |       | + APIPOD=\n    logger.go:42: 09:35:31 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175469 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.175475 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.175481 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.175486 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.175492 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.175498 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.175503 | controller |       09:35:31 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.175509 | controller |       \   logger.go:42: 09:35:31 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.175515 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.175521 | controller |       \   logger.go:42: 09:35:31 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.175527 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.175533 | controller |       \   logger.go:42: 09:35:31 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.175541 | controller |       head -1\n    logger.go:42: 09:35:31 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175557 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:31 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175566 | controller |       | + APIPOD=\n    logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175572 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.175578 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.175583 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.175589 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.175595 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.175601 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.175607 | controller |       09:35:32 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.175612 | controller |       \   logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.175619 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.175625 | controller |       \   logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.175631 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.175637 | controller |       \   logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.175642 | controller |       head -1\n    logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175648 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175653 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175659 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175664 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:44:27.175670 | controller |       \   logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:44:27.175676 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.175682 | controller |       \   logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification | error:
2026-03-14 09:44:27.175687 | controller |       unable to upgrade connection: container not found (\"watcher-api\")\n    logger.go:42:
2026-03-14 09:44:27.175693 | controller |       09:35:32 | watcher-notification/1-deploy-with-notification | ++ echo\n    logger.go:42:
2026-03-14 09:44:27.175699 | controller |       09:35:32 | watcher-notification/1-deploy-with-notification | + '[' 0 == 1 ']'\n
2026-03-14 09:44:27.175704 | controller |       \   logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification | running
2026-03-14 09:44:27.175710 | controller |       command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc
2026-03-14 09:44:27.175716 | controller |       get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:44:27.175721 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.175727 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep
2026-03-14 09:44:27.175732 | controller |       -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.175738 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.175746 | controller |       09:35:34 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.175752 | controller |       \   logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.175758 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.175764 | controller |       \   logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.175770 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.175780 | controller |       \   logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.175785 | controller |       head -1\n    logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175791 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175796 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175802 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.175816 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:44:27.175823 | controller |       \   logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:44:27.175829 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.175835 | controller |       \   logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.175841 | controller |       echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.175847 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.175853 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.175859 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_3add:c5d8ab0e913be98c16eb7b7c5c5a1976@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.175865 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.175871 | controller |       driver = messagingv2 transport_url = 'rabbit://default_user_YcxtplLsptl7k25cAVF:F735-UPJOup8HqlYaZyKNnsXxG0GtseU@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.175877 | controller |       '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.175883 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.175889 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:44:27.175895 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:44:27.175901 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.175907 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.175913 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.175919 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.175925 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.175930 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.175939 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.175945 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.175951 | controller |       tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:44:27.175957 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]'
2026-03-14 09:44:27.175962 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.175968 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.175974 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.175980 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.175985 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.175991 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.175997 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.176003 | controller |       period = 900\n    logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176008 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:35:35 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176012 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.176016 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.176021 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.176025 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.176029 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.176041 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.176045 | controller |       09:35:35 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.176050 | controller |       \   logger.go:42: 09:35:35 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.176054 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.176058 | controller |       \   logger.go:42: 09:35:35 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.176063 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.176067 | controller |       \   logger.go:42: 09:35:35 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.176071 | controller |       head -1\n    logger.go:42: 09:35:35 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176075 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:35 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176080 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:35:35 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176084 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:35:35 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176088 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:44:27.176092 | controller |       \   logger.go:42: 09:35:35 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:44:27.176098 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.176103 | controller |       \   logger.go:42: 09:35:36 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.176107 | controller |       echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.176111 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.176116 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.176120 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_3add:c5d8ab0e913be98c16eb7b7c5c5a1976@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.176124 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.176129 | controller |       driver = messagingv2 transport_url = 'rabbit://default_user_YcxtplLsptl7k25cAVF:F735-UPJOup8HqlYaZyKNnsXxG0GtseU@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.176133 | controller |       '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.176137 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.176141 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:44:27.176146 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:44:27.176153 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.176157 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.176161 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.176167 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.176171 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.176176 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.176180 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.176184 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.176188 | controller |       tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:44:27.176193 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]'
2026-03-14 09:44:27.176197 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.176201 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.176205 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.176210 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.176221 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.176225 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.176230 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.176234 | controller |       period = 900\n    logger.go:42: 09:35:36 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176240 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176246 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.176250 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.176254 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.176259 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.176263 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.176267 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.176271 | controller |       09:35:37 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.176276 | controller |       \   logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.176280 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.176284 | controller |       \   logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.176307 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.176316 | controller |       \   logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.176321 | controller |       head -1\n    logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176325 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176329 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176334 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176338 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:44:27.176344 | controller |       \   logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:44:27.176349 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.176353 | controller |       \   logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.176357 | controller |       echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.176362 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.176366 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.176370 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_3add:c5d8ab0e913be98c16eb7b7c5c5a1976@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.176374 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.176379 | controller |       driver = messagingv2 transport_url = 'rabbit://default_user_YcxtplLsptl7k25cAVF:F735-UPJOup8HqlYaZyKNnsXxG0GtseU@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.176383 | controller |       '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.176387 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.176392 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:44:27.176398 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:44:27.176402 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.176406 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.176410 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.176421 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.176426 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.176430 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.176435 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.176439 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.176443 | controller |       tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:44:27.176447 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]'
2026-03-14 09:44:27.176452 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.176456 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.176460 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.176464 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.176469 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.176473 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.176477 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.176481 | controller |       period = 900\n    logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176486 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:35:38 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176492 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.176497 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.176501 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.176505 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.176509 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.176514 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.176518 | controller |       09:35:38 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.176522 | controller |       \   logger.go:42: 09:35:38 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.176526 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.176530 | controller |       \   logger.go:42: 09:35:38 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.176535 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.176539 | controller |       \   logger.go:42: 09:35:38 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.176547 | controller |       head -1\n    logger.go:42: 09:35:38 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176551 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:39 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176556 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:35:39 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176560 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:35:39 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176565 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:44:27.176569 | controller |       \   logger.go:42: 09:35:39 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:44:27.176573 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.176578 | controller |       \   logger.go:42: 09:35:39 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.176582 | controller |       echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.176586 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.176591 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.176595 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_3add:c5d8ab0e913be98c16eb7b7c5c5a1976@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.176605 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.176609 | controller |       driver = messagingv2 transport_url = 'rabbit://default_user_YcxtplLsptl7k25cAVF:F735-UPJOup8HqlYaZyKNnsXxG0GtseU@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.176614 | controller |       '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.176618 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.176622 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:44:27.176627 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:44:27.176631 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.176635 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.176639 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.176644 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.176648 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.176652 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.176656 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.176661 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.176665 | controller |       tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:44:27.176669 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]'
2026-03-14 09:44:27.176673 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.176679 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.176685 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.176689 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.176694 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.176698 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.176702 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.176706 | controller |       period = 900\n    logger.go:42: 09:35:39 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176711 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176715 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.176719 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.176723 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.176727 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.176732 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.176736 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.176740 | controller |       09:35:40 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.176744 | controller |       \   logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.176749 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.176753 | controller |       \   logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.176757 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.176761 | controller |       \   logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.176767 | controller |       head -1\n    logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176771 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176775 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176784 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.176790 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:44:27.176798 | controller |       \   logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:44:27.176813 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.176819 | controller |       \   logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.176825 | controller |       echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.176830 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.176836 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.176841 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_3add:c5d8ab0e913be98c16eb7b7c5c5a1976@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.176850 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.176856 | controller |       driver = messagingv2 transport_url = 'rabbit://default_user_YcxtplLsptl7k25cAVF:F735-UPJOup8HqlYaZyKNnsXxG0GtseU@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.176862 | controller |       '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.176867 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.176873 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:44:27.176878 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:44:27.176884 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.176922 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.176928 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.176932 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.176936 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.176940 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.176944 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.176949 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.176953 | controller |       tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:44:27.176957 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]'
2026-03-14 09:44:27.176961 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.176965 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.176969 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.176974 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.176978 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.176982 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.176988 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.176999 | controller |       period = 900\n    logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177012 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:35:41 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177018 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.177024 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.177030 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.177036 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.177051 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.177063 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.177069 | controller |       09:35:41 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.177074 | controller |       \   logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.177080 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.177085 | controller |       \   logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.177091 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.177096 | controller |       \   logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.177101 | controller |       head -1\n    logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177107 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177113 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177118 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177124 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:44:27.177130 | controller |       \   logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:44:27.177136 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.177142 | controller |       \   logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.177148 | controller |       echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.177153 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.177159 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.177164 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_3add:c5d8ab0e913be98c16eb7b7c5c5a1976@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.177168 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.177172 | controller |       driver = messagingv2 transport_url = 'rabbit://default_user_YcxtplLsptl7k25cAVF:F735-UPJOup8HqlYaZyKNnsXxG0GtseU@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.177176 | controller |       '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.177181 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.177185 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:44:27.177189 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:44:27.177193 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.177197 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.177201 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.177206 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.177218 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.177236 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.177243 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.177249 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.177254 | controller |       tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:44:27.177258 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]'
2026-03-14 09:44:27.177262 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.177266 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.177280 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.177285 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.177311 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.177316 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.177320 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.177328 | controller |       period = 900\n    logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177332 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177336 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.177340 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.177344 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.177349 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.177353 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.177357 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.177361 | controller |       09:35:43 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.177365 | controller |       \   logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.177369 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.177374 | controller |       \   logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.177378 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.177382 | controller |       \   logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.177386 | controller |       head -1\n    logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177390 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177394 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177399 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177403 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:44:27.177409 | controller |       \   logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:44:27.177415 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.177419 | controller |       \   logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.177424 | controller |       echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.177428 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.177432 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.177436 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_3add:c5d8ab0e913be98c16eb7b7c5c5a1976@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.177441 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.177445 | controller |       driver = messagingv2 transport_url = 'rabbit://default_user_YcxtplLsptl7k25cAVF:F735-UPJOup8HqlYaZyKNnsXxG0GtseU@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.177449 | controller |       '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.177453 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.177458 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:44:27.177462 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:44:27.177471 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.177484 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.177489 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.177493 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.177497 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.177501 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.177506 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.177510 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.177514 | controller |       tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:44:27.177518 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]'
2026-03-14 09:44:27.177522 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.177527 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.177531 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.177535 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.177539 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.177543 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.177547 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.177553 | controller |       period = 900\n    logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177557 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:35:44 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177562 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.177567 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.177571 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.177576 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.177580 | controller |       |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=')
2026-03-14 09:44:27.177584 | controller |       == 1 ]\n        else\n          exit 1\n        fi\n        ]\n    logger.go:42:
2026-03-14 09:44:27.177588 | controller |       09:35:44 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n
2026-03-14 09:44:27.177592 | controller |       \   logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification | Already
2026-03-14 09:44:27.177596 | controller |       on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.177600 | controller |       \   logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.177605 | controller |       oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.177609 | controller |       \   logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.177613 | controller |       head -1\n    logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177619 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177624 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177661 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177670 | controller |       | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n
2026-03-14 09:44:27.177676 | controller |       \   logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification | +++
2026-03-14 09:44:27.177682 | controller |       oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.177688 | controller |       \   logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification | ++
2026-03-14 09:44:27.177694 | controller |       echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.177700 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.177706 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.177722 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_3add:c5d8ab0e913be98c16eb7b7c5c5a1976@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.177727 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.177732 | controller |       driver = messagingv2 transport_url = 'rabbit://default_user_YcxtplLsptl7k25cAVF:F735-UPJOup8HqlYaZyKNnsXxG0GtseU@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.177736 | controller |       '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.177740 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.177748 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:44:27.177752 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:44:27.177756 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.177761 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.177765 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.177769 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.177773 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.177777 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.177781 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.177786 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.177790 | controller |       tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:44:27.177794 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]'
2026-03-14 09:44:27.177798 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.177803 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.177807 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.177811 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.177815 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.177819 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.177827 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.177831 | controller |       period = 900\n    logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177835 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification
2026-03-14 09:44:27.177840 | controller |       | test step completed 1-deploy-with-notification\n    logger.go:42: 09:35:45 | watcher-notification/2-cleanup-watcher
2026-03-14 09:44:27.177844 | controller |       | starting test step 2-cleanup-watcher\n    logger.go:42: 09:35:51 | watcher-notification/2-cleanup-watcher
2026-03-14 09:44:27.177849 | controller |       | test step completed 2-cleanup-watcher\n    logger.go:42: 09:35:51 | watcher-notification
2026-03-14 09:44:27.177854 | controller |       | skipping kubernetes event logging\n=== CONT  kuttl/harness/watcher-topology\n
2026-03-14 09:44:27.177858 | controller |       \   logger.go:42: 09:35:51 | watcher-topology | Skipping creation of user-supplied
2026-03-14 09:44:27.177862 | controller |       namespace: watcher-kuttl-default\n    logger.go:42: 09:35:51 | watcher-topology/0-cleanup-watcher
2026-03-14 09:44:27.177866 | controller |       | starting test step 0-cleanup-watcher\n    logger.go:42: 09:35:51 | watcher-topology/0-cleanup-watcher
2026-03-14 09:44:27.177870 | controller |       | test step completed 0-cleanup-watcher\n    logger.go:42: 09:35:51 | watcher-topology/1-deploy-with-topology
2026-03-14 09:44:27.177875 | controller |       | starting test step 1-deploy-with-topology\n    logger.go:42: 09:35:51 | watcher-topology/1-deploy-with-topology
2026-03-14 09:44:27.177879 | controller |       | Topology:watcher-kuttl-default/watcher-api created\n    logger.go:42: 09:35:51
2026-03-14 09:44:27.177885 | controller |       | watcher-topology/1-deploy-with-topology | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:44:27.177889 | controller |       created\n    logger.go:42: 09:36:15 | watcher-topology/1-deploy-with-topology |
2026-03-14 09:44:27.177899 | controller |       test step completed 1-deploy-with-topology\n    logger.go:42: 09:36:15 | watcher-topology/2-cleanup-watcher
2026-03-14 09:44:27.177904 | controller |       | starting test step 2-cleanup-watcher\n    logger.go:42: 09:36:15 | watcher-topology/2-cleanup-watcher
2026-03-14 09:44:27.177908 | controller |       | running command: [sh -c set -ex\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.177912 | controller |       -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0
2026-03-14 09:44:27.177916 | controller |       ]\n        ]\n    logger.go:42: 09:36:15 | watcher-topology/2-cleanup-watcher |
2026-03-14 09:44:27.177920 | controller |       + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.177925 | controller |       value -c Name -c Type\n    logger.go:42: 09:36:15 | watcher-topology/2-cleanup-watcher
2026-03-14 09:44:27.177930 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:36:18 | watcher-topology/2-cleanup-watcher
2026-03-14 09:44:27.177934 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:36:19 | watcher-topology/2-cleanup-watcher
2026-03-14 09:44:27.177939 | controller |       | running command: [sh -c set -ex\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.177943 | controller |       -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0
2026-03-14 09:44:27.177947 | controller |       ]\n        ]\n    logger.go:42: 09:36:19 | watcher-topology/2-cleanup-watcher |
2026-03-14 09:44:27.177951 | controller |       + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.177955 | controller |       value -c Name -c Type\n    logger.go:42: 09:36:19 | watcher-topology/2-cleanup-watcher
2026-03-14 09:44:27.177959 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:36:21 | watcher-topology/2-cleanup-watcher
2026-03-14 09:44:27.177964 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:36:22 | watcher-topology/2-cleanup-watcher
2026-03-14 09:44:27.177968 | controller |       | running command: [sh -c set -ex\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.177972 | controller |       -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0
2026-03-14 09:44:27.177976 | controller |       ]\n        ]\n    logger.go:42: 09:36:22 | watcher-topology/2-cleanup-watcher |
2026-03-14 09:44:27.177980 | controller |       + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.177984 | controller |       value -c Name -c Type\n    logger.go:42: 09:36:22 | watcher-topology/2-cleanup-watcher
2026-03-14 09:44:27.177988 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:36:24 | watcher-topology/2-cleanup-watcher
2026-03-14 09:44:27.177993 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:36:24 | watcher-topology/2-cleanup-watcher
2026-03-14 09:44:27.177997 | controller |       | test step completed 2-cleanup-watcher\n    logger.go:42: 09:36:24 | watcher-topology
2026-03-14 09:44:27.178001 | controller |       | skipping kubernetes event logging\n=== CONT  kuttl/harness/watcher-tls-certs-change\n
2026-03-14 09:44:27.178005 | controller |       \   logger.go:42: 09:36:24 | watcher-tls-certs-change | Skipping creation of user-supplied
2026-03-14 09:44:27.178009 | controller |       namespace: watcher-kuttl-default\n    logger.go:42: 09:36:24 | watcher-tls-certs-change/0-cleanup-watcher
2026-03-14 09:44:27.178013 | controller |       | starting test step 0-cleanup-watcher\n    logger.go:42: 09:36:24 | watcher-tls-certs-change/0-cleanup-watcher
2026-03-14 09:44:27.178018 | controller |       | test step completed 0-cleanup-watcher\n    logger.go:42: 09:36:24 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.178022 | controller |       | starting test step 1-deploy-with-tlse\n    logger.go:42: 09:36:24 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.178026 | controller |       | Certificate:watcher-kuttl-default/watcher-internal-svc created\n    logger.go:42:
2026-03-14 09:44:27.178032 | controller |       09:36:24 | watcher-tls-certs-change/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc
2026-03-14 09:44:27.178036 | controller |       created\n    logger.go:42: 09:36:24 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.178040 | controller |       | Watcher:watcher-kuttl-default/watcher-kuttl created\n    logger.go:42: 09:36:25
2026-03-14 09:44:27.178044 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo
2026-03-14 09:44:27.178049 | controller |       pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.178053 | controller |       cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc get -n $NAMESPACE
2026-03-14 09:44:27.178057 | controller |       secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n
2026-03-14 09:44:27.178061 | controller |       \       # ensure that the svc secret and cert secret match\n        if [ \"${public_svc_cert}\"
2026-03-14 09:44:27.178065 | controller |       != \"${public_secret_cert}\" ]; then\n          exit 1\n        fi\n        \n        internal_svc_cert=$(oc
2026-03-14 09:44:27.178069 | controller |       rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n
2026-03-14 09:44:27.178074 | controller |       \       internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.178078 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:44:27.178087 | controller |       secret and cert secret match\n        if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\"
2026-03-14 09:44:27.178092 | controller |       ]; then\n          exit 1\n        fi\n        ]\n    logger.go:42: 09:36:25 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.178096 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:44:27.178100 | controller |       \   logger.go:42: 09:36:25 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.178105 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:44:27.178109 | controller |       09:36:25 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:44:27.178113 | controller |       09:36:26 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:44:27.178117 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178121 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:44:27.178125 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178129 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178137 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178141 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178145 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:44:27.178149 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178153 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178158 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178162 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:36:26 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.178166 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:44:27.178170 | controller |       \   logger.go:42: 09:36:26 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.178174 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:44:27.178181 | controller |       09:36:26 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:44:27.178185 | controller |       09:36:27 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:44:27.178189 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178194 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:44:27.178198 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178202 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178206 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178210 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178214 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:44:27.178219 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178223 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178227 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178231 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:36:27 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.178235 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:44:27.178239 | controller |       \   logger.go:42: 09:36:27 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.178244 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:44:27.178248 | controller |       09:36:27 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:44:27.178252 | controller |       09:36:28 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:44:27.178256 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178260 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:44:27.178265 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178269 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178278 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178283 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178287 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:44:27.178309 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178314 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178318 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178322 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:36:28 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.178326 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:44:27.178330 | controller |       \   logger.go:42: 09:36:28 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.178335 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:44:27.178339 | controller |       09:36:28 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:44:27.178345 | controller |       09:36:29 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:44:27.178350 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178354 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:44:27.178360 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178364 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178368 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178372 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178376 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:44:27.178381 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178385 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178389 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178393 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:36:29 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.178398 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:44:27.178402 | controller |       \   logger.go:42: 09:36:30 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.178406 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:44:27.178412 | controller |       09:36:30 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:44:27.178416 | controller |       09:36:31 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:44:27.178420 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178425 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:44:27.178429 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178433 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178437 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178442 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178446 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:44:27.178450 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178454 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178458 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178463 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:36:31 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.178467 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:44:27.178471 | controller |       \   logger.go:42: 09:36:31 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.178477 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:44:27.178481 | controller |       09:36:31 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:44:27.178485 | controller |       09:36:32 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:44:27.178498 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178503 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:44:27.178507 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178511 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178516 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178520 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178524 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:44:27.178528 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178533 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178537 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178541 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:36:32 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.178545 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:44:27.178550 | controller |       \   logger.go:42: 09:36:32 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.178554 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:44:27.178558 | controller |       09:36:32 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:44:27.178562 | controller |       09:36:33 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:44:27.178568 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178572 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:44:27.178577 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178581 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178585 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178589 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178594 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:44:27.178598 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178602 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178606 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178611 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:36:33 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.178616 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:44:27.178621 | controller |       \   logger.go:42: 09:36:33 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.178627 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:44:27.178633 | controller |       09:36:33 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:44:27.178639 | controller |       09:36:34 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:44:27.178645 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178651 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:44:27.178659 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178663 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178667 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178671 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178675 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:44:27.178680 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178684 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178688 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178692 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:36:34 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.178703 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:44:27.178707 | controller |       \   logger.go:42: 09:36:34 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.178711 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:44:27.178716 | controller |       09:36:34 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:44:27.178720 | controller |       09:36:36 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:44:27.178724 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178728 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:44:27.178732 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178736 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178741 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178745 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178749 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:44:27.178753 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178757 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178762 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178766 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:36:36 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.178770 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:44:27.178774 | controller |       \   logger.go:42: 09:36:36 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.178781 | controller |       from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n    logger.go:42:
2026-03-14 09:44:27.178785 | controller |       09:36:36 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n    logger.go:42:
2026-03-14 09:44:27.178790 | controller |       09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c
2026-03-14 09:44:27.178794 | controller |       set -euxo pipefail\n        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178798 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        public_secret_cert=$(oc
2026-03-14 09:44:27.178802 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178808 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178812 | controller |       \       if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178816 | controller |       1\n        fi\n        \n        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api
2026-03-14 09:44:27.178820 | controller |       watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        internal_secret_cert=$(oc
2026-03-14 09:44:27.178825 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.178829 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.178833 | controller |       \       if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n          exit
2026-03-14 09:44:27.178837 | controller |       1\n        fi\n        ]\n    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.178841 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:44:27.178845 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN
2026-03-14 09:44:27.178850 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.178854 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.178858 | controller |       09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.178862 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.178867 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.178871 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.178875 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.178885 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.178890 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.178894 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.178900 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.178904 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.178908 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.178913 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.178917 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.178921 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.178925 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.178931 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.178935 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.178939 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.178944 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.178948 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.178952 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.178957 | controller |       | ++ base64 --decode\n    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.178977 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:44:27.178982 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN
2026-03-14 09:44:27.178986 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.178990 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.178994 | controller |       09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.178998 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.179003 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.179009 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.179015 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.179021 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.179030 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.179036 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.179042 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.179055 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.179060 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.179064 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.179069 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.179073 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.179080 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.179085 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.179089 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.179093 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.179098 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.179102 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.179106 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.179111 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.179115 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.179119 | controller |       09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.179124 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.179128 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.179132 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.179136 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.179141 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.179145 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.179149 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.179153 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.179158 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.179166 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.179170 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.179174 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.179178 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.179183 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.179194 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.179199 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.179203 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.179207 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.179212 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.179216 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:37
2026-03-14 09:44:27.179220 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:44:27.179225 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.179229 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.179233 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.179237 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.179243 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.179252 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.179267 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.179274 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.179280 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.179286 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.179315 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.179322 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.179328 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.179338 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.179344 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.179350 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.179356 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.179362 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.179371 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.179377 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.179383 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.179387 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:44:27.179391 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN
2026-03-14 09:44:27.179403 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.179408 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.179412 | controller |       09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.179417 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.179421 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.179425 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.179429 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.179433 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.179438 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.179442 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.179446 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.179450 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.179454 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.179458 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.179463 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.179467 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.179471 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.179475 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.179479 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.179488 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.179493 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.179497 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.179501 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.179505 | controller |       | ++ base64 --decode\n    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.179510 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:44:27.179514 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN
2026-03-14 09:44:27.179518 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.179522 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.179526 | controller |       09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.179531 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.179535 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.179546 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.179552 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.179558 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.179564 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.179570 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.179576 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.179582 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.179588 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.179593 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.179599 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.179604 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.179609 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.179613 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.179618 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.179627 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.179633 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.179639 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.179645 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.179651 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.179657 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.179663 | controller |       09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.179669 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.179678 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.179683 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.179689 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.179695 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.179701 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.179707 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.179713 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.179726 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.179733 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.179739 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.179745 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.179750 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.179756 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.179762 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.179768 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.179774 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.179782 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.179789 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.179795 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:37
2026-03-14 09:44:27.179801 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:44:27.179807 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.179813 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.179818 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.179824 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.179833 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.179839 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.179845 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.179851 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.179856 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.179862 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.179868 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.179874 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.179880 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.179885 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.179893 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.179899 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.179911 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.179917 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.179923 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.179929 | controller |       \   logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.179937 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.179943 | controller |       | running command: [sh -c set -euxo pipefail\n        public_svc_cert=$(oc rsh -n
2026-03-14 09:44:27.179949 | controller |       $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n
2026-03-14 09:44:27.179955 | controller |       \       public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.179959 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:44:27.179963 | controller |       secret and cert secret match\n        if [ \"${public_svc_cert}\" != \"${public_secret_cert}\"
2026-03-14 09:44:27.179967 | controller |       ]; then\n          exit 1\n        fi\n        \n        internal_svc_cert=$(oc
2026-03-14 09:44:27.179972 | controller |       rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n
2026-03-14 09:44:27.179978 | controller |       \       internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.179984 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:44:27.179990 | controller |       secret and cert secret match\n        if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\"
2026-03-14 09:44:27.179996 | controller |       ]; then\n          exit 1\n        fi\n        ]\n    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.180002 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:44:27.180008 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN
2026-03-14 09:44:27.180014 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.180019 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.180025 | controller |       09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.180031 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.180036 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.180042 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.180048 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.180054 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.180060 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.180069 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.180075 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.180081 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.180087 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.180093 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.180101 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.180114 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.180120 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.180126 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.180132 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.180138 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.180144 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.180150 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.180156 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.180161 | controller |       | ++ base64 --decode\n    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.180167 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:44:27.180173 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN
2026-03-14 09:44:27.180179 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.180184 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.180190 | controller |       09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.180196 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.180202 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.180208 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.180213 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.180219 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.180225 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.180231 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.180237 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.180242 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.180248 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.180261 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.180266 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.180272 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.180278 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.180284 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.180308 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.180313 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.180323 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.180328 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.180332 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.180336 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.180340 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.180345 | controller |       09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.180349 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.180353 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.180357 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.180361 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.180366 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.180370 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.180374 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.180378 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.180382 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.180387 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.180391 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.180395 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.180401 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.180405 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.180412 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.180416 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.180421 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.180425 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.180429 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.180433 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:39
2026-03-14 09:44:27.180438 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:44:27.180442 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.180446 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.180450 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.180459 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.180464 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.180468 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.180472 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.180476 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.180481 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.180485 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.180489 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.180493 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.180497 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.180502 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.180506 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.180511 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.180516 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.180520 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.180526 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.180531 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.180535 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.180539 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:44:27.180544 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN
2026-03-14 09:44:27.180548 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.180552 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.180556 | controller |       09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.180561 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.180565 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.180569 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.180573 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.180577 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.180582 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.180586 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.180594 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.180599 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.180603 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.180607 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.180611 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.180616 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.180620 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.180626 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.180630 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.180634 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.180638 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.180643 | controller |       \   logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.180647 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.180651 | controller |       | ++ base64 --decode\n    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.180655 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:44:27.180660 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN
2026-03-14 09:44:27.180664 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.180668 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.180672 | controller |       09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.180676 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.180681 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.180685 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.180691 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.180696 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.180700 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.180704 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.180708 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.180712 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.180717 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.180721 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.180729 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.180735 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.180743 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.180749 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.180755 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.180762 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.180767 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.180773 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.180779 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.180785 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.180791 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.180797 | controller |       09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.180804 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.180810 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.180815 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.180819 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.180824 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.180828 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.180835 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.180839 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.180845 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.180851 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.180857 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.180863 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.180868 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.180874 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.180883 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.180889 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.180895 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.180906 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.180911 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.180917 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:40
2026-03-14 09:44:27.180922 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:44:27.180926 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.180931 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.180936 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.180943 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.180949 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.180955 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.180960 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.180966 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.180972 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.180981 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.180987 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.180993 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.180999 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.181004 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.181008 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.181013 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.181018 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.181024 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.181028 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.181033 | controller |       \   logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.181046 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.181054 | controller |       | running command: [sh -c set -euxo pipefail\n        public_svc_cert=$(oc rsh -n
2026-03-14 09:44:27.181060 | controller |       $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n
2026-03-14 09:44:27.181066 | controller |       \       public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.181072 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:44:27.181077 | controller |       secret and cert secret match\n        if [ \"${public_svc_cert}\" != \"${public_secret_cert}\"
2026-03-14 09:44:27.181083 | controller |       ]; then\n          exit 1\n        fi\n        \n        internal_svc_cert=$(oc
2026-03-14 09:44:27.181089 | controller |       rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n
2026-03-14 09:44:27.181101 | controller |       \       internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.181107 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:44:27.181113 | controller |       secret and cert secret match\n        if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\"
2026-03-14 09:44:27.181118 | controller |       ]; then\n          exit 1\n        fi\n        ]\n    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.181123 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:44:27.181127 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN
2026-03-14 09:44:27.181133 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.181139 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.181145 | controller |       09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.181151 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.181157 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.181163 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.181169 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.181175 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.181180 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.181186 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.181192 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.181203 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.181208 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.181214 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.181223 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.181229 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.181235 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.181241 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.181247 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.181252 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.181258 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.181264 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.181270 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.181276 | controller |       | ++ base64 --decode\n    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.181282 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:44:27.181303 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN
2026-03-14 09:44:27.181324 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.181331 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.181337 | controller |       09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.181343 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.181349 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.181355 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.181361 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.181367 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.181372 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.181378 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.181387 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.181392 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.181398 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.181404 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.181410 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.181416 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.181422 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.181428 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.181437 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.181443 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.181449 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.181454 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.181460 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.181466 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.181472 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.181480 | controller |       09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.181485 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.181491 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.181497 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.181503 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.181516 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.181523 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.181528 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.181534 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.181540 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.181549 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.181555 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.181561 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.181566 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.181572 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.181578 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.181584 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.181590 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.181596 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.181602 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.181608 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:41
2026-03-14 09:44:27.181616 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:44:27.181622 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.181628 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.181635 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.181684 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.181690 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.181695 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.181700 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.181704 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.181708 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.181712 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.181716 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.181721 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.181736 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.181741 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.181745 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.181749 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.181753 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.181757 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.181762 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.181766 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.181770 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.181774 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:44:27.181779 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN
2026-03-14 09:44:27.181783 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.181787 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.181791 | controller |       09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.181796 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.181800 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.181804 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.181808 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.181812 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.181819 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.181823 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.181827 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.181831 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.181836 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.181840 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.181845 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.181850 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.181854 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.181858 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.181867 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.181872 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.181876 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.181881 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.181885 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.181889 | controller |       | ++ base64 --decode\n    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.181893 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:44:27.181898 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN
2026-03-14 09:44:27.181902 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.181906 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.181911 | controller |       09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.181915 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.181921 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.181926 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.181930 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.181935 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.181940 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.181945 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.181949 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.181955 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.181960 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.181965 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.181969 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.181974 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.181978 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.181982 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.181986 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.181990 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.181994 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.181999 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.182003 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.182012 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.182033 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.182038 | controller |       09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.182042 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.182046 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.182050 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.182055 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.182059 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.182063 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.182067 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.182071 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.182076 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.182080 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.182084 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.182092 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.182096 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.182101 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.182105 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.182109 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.182113 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.182117 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.182122 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.182126 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:41
2026-03-14 09:44:27.182130 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:44:27.182134 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.182138 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.182143 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.182147 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.182151 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.182155 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.182165 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.182169 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.182173 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.182178 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.182182 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.182186 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.182190 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.182194 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.182203 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.182207 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.182211 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.182216 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.182220 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.182224 | controller |       \   logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.182228 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:36:42 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.182233 | controller |       | running command: [sh -c set -euxo pipefail\n        public_svc_cert=$(oc rsh -n
2026-03-14 09:44:27.182237 | controller |       $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n
2026-03-14 09:44:27.182241 | controller |       \       public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.182245 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:44:27.182249 | controller |       secret and cert secret match\n        if [ \"${public_svc_cert}\" != \"${public_secret_cert}\"
2026-03-14 09:44:27.182253 | controller |       ]; then\n          exit 1\n        fi\n        \n        internal_svc_cert=$(oc
2026-03-14 09:44:27.182258 | controller |       rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n
2026-03-14 09:44:27.182262 | controller |       \       internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.182266 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:44:27.182270 | controller |       secret and cert secret match\n        if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\"
2026-03-14 09:44:27.182274 | controller |       ]; then\n          exit 1\n        fi\n        ]\n    logger.go:42: 09:36:42 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.182278 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:44:27.182283 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN
2026-03-14 09:44:27.182287 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.182306 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.182310 | controller |       09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.182315 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.182319 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.182328 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.182332 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.182336 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.182342 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.182347 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.182351 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.182355 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.182359 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.182363 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.182368 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.182372 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.182376 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.182380 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.182384 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.182390 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.182395 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.182399 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.182403 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.182408 | controller |       | ++ base64 --decode\n    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.182412 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:44:27.182416 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN
2026-03-14 09:44:27.182420 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.182424 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.182429 | controller |       09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.182433 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.182437 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.182441 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.182445 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.182450 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.182455 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.182464 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.182469 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.182473 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.182477 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.182481 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.182485 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.182490 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.182494 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.182498 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.182502 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.182506 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.182511 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.182515 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.182521 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.182525 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.182529 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.182533 | controller |       09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.182537 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.182542 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.182546 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.182552 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.182556 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.182560 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.182566 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.182570 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.182574 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.182578 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.182583 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.182587 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.182591 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.182600 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.182604 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.182608 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.182613 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.182617 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.182621 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.182625 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:43
2026-03-14 09:44:27.182630 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:44:27.182634 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.182638 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.182642 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.182647 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.182651 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.182655 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.182659 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.182665 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.182669 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.182675 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.182679 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.182683 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.182688 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.182693 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.182697 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.182702 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.182706 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.182710 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.182714 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.182718 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.182723 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.182731 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:44:27.182736 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN
2026-03-14 09:44:27.182740 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.182745 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.182749 | controller |       09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.182753 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.182757 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.182762 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.182766 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.182770 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.182774 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.182779 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.182783 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.182789 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.182793 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.182799 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.182803 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.182808 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.182812 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.182816 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.182820 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.182824 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.182829 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.182833 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.182837 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.182841 | controller |       | ++ base64 --decode\n    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.182846 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:44:27.182850 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN
2026-03-14 09:44:27.182854 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.182858 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.182862 | controller |       09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.182871 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.182876 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.182881 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.182885 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.182889 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.182894 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.182898 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.182904 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.182908 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.182912 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.182916 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.182921 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.182925 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.182931 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.182935 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.182939 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.182943 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.182948 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.182952 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.182956 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.182960 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.182965 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.182969 | controller |       09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.182973 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.182977 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.182982 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.182986 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.182990 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.182994 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.182998 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.183007 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.183014 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.183019 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.183023 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.183027 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.183031 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.183036 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.183040 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.183044 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.183050 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.183054 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.183058 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.183064 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:43
2026-03-14 09:44:27.183068 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:44:27.183073 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.183077 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.183081 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.183085 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.183089 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.183094 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.183098 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.183102 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.183106 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.183110 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.183115 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.183120 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.183124 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.183129 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.183137 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.183142 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.183146 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.183150 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.183155 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.183160 | controller |       \   logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.183165 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.183169 | controller |       | running command: [sh -c set -euxo pipefail\n        public_svc_cert=$(oc rsh -n
2026-03-14 09:44:27.183173 | controller |       $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n
2026-03-14 09:44:27.183177 | controller |       \       public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.183182 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:44:27.183186 | controller |       secret and cert secret match\n        if [ \"${public_svc_cert}\" != \"${public_secret_cert}\"
2026-03-14 09:44:27.183190 | controller |       ]; then\n          exit 1\n        fi\n        \n        internal_svc_cert=$(oc
2026-03-14 09:44:27.183194 | controller |       rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n
2026-03-14 09:44:27.183198 | controller |       \       internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.183203 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:44:27.183207 | controller |       secret and cert secret match\n        if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\"
2026-03-14 09:44:27.183211 | controller |       ]; then\n          exit 1\n        fi\n        ]\n    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.183215 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:44:27.183220 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN
2026-03-14 09:44:27.183224 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.183228 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.183232 | controller |       09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.183238 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.183242 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.183247 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.183252 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.183257 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.183261 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.183265 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.183269 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.183273 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.183278 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.183282 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.183302 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.183310 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.183314 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.183318 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.183323 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.183327 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.183331 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.183335 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.183340 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.183344 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:44:27.183348 | controller |       \   logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:44:27.183352 | controller |       --decode\n    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.183356 | controller |       | + public_secret_cert='-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:45
2026-03-14 09:44:27.183361 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:44:27.183365 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.183369 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.183373 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.183379 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.183385 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.183389 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.183394 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.183398 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.183402 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.183406 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.183410 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.183415 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.183419 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.183423 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.183427 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.183431 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.183441 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.183446 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.183450 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.183454 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.183459 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.183463 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.183467 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.183471 | controller |       09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.183476 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.183480 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.183484 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.183490 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.183494 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.183498 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.183503 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.183509 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.183513 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.183517 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.183521 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.183526 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.183530 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.183534 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.183538 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.183543 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.183547 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.183551 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.183555 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.183560 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:45
2026-03-14 09:44:27.183564 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:44:27.183568 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.183577 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.183581 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.183585 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.183590 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.183594 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.183599 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.183604 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.183608 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.183612 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.183617 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.183626 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.183632 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.183638 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.183644 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.183650 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.183655 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.183659 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.183664 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.183668 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.183672 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.183677 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:44:27.183682 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN
2026-03-14 09:44:27.183686 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.183690 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.183694 | controller |       09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.183699 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.183703 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.183707 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.183711 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.183715 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.183727 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.183732 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.183736 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.183740 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.183744 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.183748 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.183753 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.183757 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.183761 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.183767 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.183772 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.183776 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.183781 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.183785 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.183789 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.183794 | controller |       | ++ base64 --decode\n    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.183798 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:44:27.183802 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN
2026-03-14 09:44:27.183806 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.183810 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.183815 | controller |       09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.183819 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.183823 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.183827 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.183832 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.183839 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.183843 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.183847 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.183851 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.183855 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.183864 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.183869 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.183873 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.183877 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.183882 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.183886 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.183890 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.183894 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.183900 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.183905 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.183909 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.183914 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.183919 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.183924 | controller |       09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.183928 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.183933 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.183938 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.183943 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.183947 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.183952 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.183957 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.183961 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.183966 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.183970 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.183974 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.183978 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.183982 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.183987 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.183991 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.183999 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.184004 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.184008 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.184012 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.184017 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:45
2026-03-14 09:44:27.184021 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:44:27.184025 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.184029 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.184036 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.184040 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.184044 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.184048 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.184053 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.184057 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.184061 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.184067 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.184071 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.184075 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.184079 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.184083 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.184088 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.184092 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.184096 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.184100 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.184104 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.184109 | controller |       \   logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.184113 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.184117 | controller |       | running command: [sh -c set -euxo pipefail\n        public_svc_cert=$(oc rsh -n
2026-03-14 09:44:27.184121 | controller |       $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n
2026-03-14 09:44:27.184126 | controller |       \       public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.184130 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:44:27.184134 | controller |       secret and cert secret match\n        if [ \"${public_svc_cert}\" != \"${public_secret_cert}\"
2026-03-14 09:44:27.184144 | controller |       ]; then\n          exit 1\n        fi\n        \n        internal_svc_cert=$(oc
2026-03-14 09:44:27.184148 | controller |       rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n
2026-03-14 09:44:27.184153 | controller |       \       internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.184157 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:44:27.184161 | controller |       secret and cert secret match\n        if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\"
2026-03-14 09:44:27.184165 | controller |       ]; then\n          exit 1\n        fi\n        ]\n    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.184169 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:44:27.184175 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN
2026-03-14 09:44:27.184179 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.184185 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.184189 | controller |       09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.184195 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.184199 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.184203 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.184207 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.184212 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.184216 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.184220 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.184224 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.184228 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.184233 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.184237 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.184241 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.184245 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.184249 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.184254 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.184258 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.184263 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.184269 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.184275 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.184283 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.184304 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:44:27.184319 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:44:27.184326 | controller |       --decode\n    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.184332 | controller |       | + public_secret_cert='-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:46
2026-03-14 09:44:27.184338 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:44:27.184345 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.184349 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.184353 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.184357 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.184362 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.184366 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.184370 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.184374 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.184381 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.184385 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.184389 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.184393 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.184398 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.184402 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.184406 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.184410 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.184414 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.184419 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.184423 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.184427 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.184431 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.184436 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.184440 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.184444 | controller |       09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.184450 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.184454 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.184463 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.184468 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.184472 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.184476 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.184481 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.184485 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.184489 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.184493 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.184498 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.184503 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.184508 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.184512 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.184516 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.184520 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.184525 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.184529 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.184533 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.184537 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:46
2026-03-14 09:44:27.184542 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:44:27.184546 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.184550 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.184554 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.184560 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.184564 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.184568 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.184573 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.184577 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.184581 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.184585 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.184593 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.184599 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.184603 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.184607 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.184611 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.184618 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.184623 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.184627 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.184631 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.184635 | controller |       \   logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.184639 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.184644 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:44:27.184648 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN
2026-03-14 09:44:27.184652 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.184656 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.184661 | controller |       09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.184665 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.184669 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.184679 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.184684 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.184688 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.184692 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.184696 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.184700 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.184704 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.184709 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.184713 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.184717 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.184721 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.184726 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.184734 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.184739 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.184743 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.184747 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.184753 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.184757 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.184762 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:44:27.184766 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:44:27.184770 | controller |       --decode\n    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.184774 | controller |       | + internal_secret_cert='-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:47
2026-03-14 09:44:27.184779 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:44:27.184783 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.184787 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.184793 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.184797 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.184801 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.184805 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.184810 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.184814 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.184818 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.184822 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.184826 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.184831 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.184835 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.184839 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.184843 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.184847 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.184852 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.184856 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.184860 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.184869 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.184873 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.184877 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.184881 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.184886 | controller |       09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.184890 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.184894 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.184900 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.184906 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.184910 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.184914 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.184918 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.184923 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.184927 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.184931 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.184935 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.184939 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.184944 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.184948 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.184952 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.184956 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.184960 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.184965 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.184969 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.184973 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:47
2026-03-14 09:44:27.184977 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:44:27.184982 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.184986 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.184990 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.184994 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.185002 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.185008 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.185014 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.185018 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.185023 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.185027 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.185031 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.185035 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.185040 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.185044 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.185048 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.185052 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.185056 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.185061 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.185065 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.185069 | controller |       \   logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.185073 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.185078 | controller |       | running command: [sh -c set -euxo pipefail\n        public_svc_cert=$(oc rsh -n
2026-03-14 09:44:27.185082 | controller |       $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n
2026-03-14 09:44:27.185088 | controller |       \       public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.185093 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:44:27.185097 | controller |       secret and cert secret match\n        if [ \"${public_svc_cert}\" != \"${public_secret_cert}\"
2026-03-14 09:44:27.185101 | controller |       ]; then\n          exit 1\n        fi\n        \n        internal_svc_cert=$(oc
2026-03-14 09:44:27.185106 | controller |       rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n
2026-03-14 09:44:27.185110 | controller |       \       internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.185114 | controller |       -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n        # ensure that the svc
2026-03-14 09:44:27.185118 | controller |       secret and cert secret match\n        if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\"
2026-03-14 09:44:27.185122 | controller |       ]; then\n          exit 1\n        fi\n        ]\n    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.185127 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:44:27.185132 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN
2026-03-14 09:44:27.185136 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.185141 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.185145 | controller |       09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.185149 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.185159 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.185164 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.185169 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.185173 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.185177 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.185181 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.185186 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.185190 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.185194 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.185198 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.185202 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.185207 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.185211 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.185215 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.185219 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.185223 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.185228 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.185233 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.185238 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.185242 | controller |       | ++ base64 --decode\n    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.185247 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:44:27.185252 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN
2026-03-14 09:44:27.185256 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.185260 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.185264 | controller |       09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.185269 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.185273 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.185277 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.185281 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.185285 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.185310 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.185315 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.185319 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.185323 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.185327 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.185332 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.185338 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.185342 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.185346 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.185351 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.185355 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.185359 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.185363 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.185367 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.185372 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.185377 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.185382 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.185388 | controller |       09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.185393 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.185397 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.185401 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.185405 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.185410 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.185414 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.185418 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.185422 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.185426 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.185430 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.185435 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.185444 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.185448 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.185453 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.185457 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.185463 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.185467 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.185471 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.185475 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.185480 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:48
2026-03-14 09:44:27.185484 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:44:27.185488 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.185494 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.185498 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.185503 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.185507 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.185511 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.185515 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.185519 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.185524 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.185528 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.185532 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.185536 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.185541 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.185545 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.185549 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.185553 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.185558 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.185562 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.185566 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.185575 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.185582 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.185588 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:44:27.185594 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN
2026-03-14 09:44:27.185600 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.185607 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.185616 | controller |       09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.185622 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.185631 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.185638 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.185642 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.185659 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.185666 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.185671 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.185676 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.185680 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.185684 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.185688 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.185693 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.185697 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.185701 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.185706 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.185710 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.185714 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.185719 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.185724 | controller |       \   logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.185728 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.185732 | controller |       | ++ base64 --decode\n    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.185737 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:44:27.185741 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN
2026-03-14 09:44:27.185745 | controller |       CERTIFICATE-----\n    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.185752 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.185762 | controller |       09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.185767 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.185771 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.185776 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.185780 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.185787 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.185791 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.185795 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.185799 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.185804 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.185808 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.185812 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.185817 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.185821 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.185825 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.185829 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.185834 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.185838 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.185842 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.185847 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.185851 | controller |       CERTIFICATE-----'\n    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.185855 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.185860 | controller |       | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.185864 | controller |       09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.185870 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.185876 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.185882 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.185888 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.185894 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.185906 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.185912 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.185922 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.185928 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.185934 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.185940 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.185946 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.185952 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.185959 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.185966 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.185972 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.185977 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.185982 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.185987 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.185991 | controller |       CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:49
2026-03-14 09:44:27.185995 | controller |       | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:44:27.185999 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx\n
2026-03-14 09:44:27.186004 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm\n
2026-03-14 09:44:27.186008 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72\n
2026-03-14 09:44:27.186014 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6\n
2026-03-14 09:44:27.186018 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr\n
2026-03-14 09:44:27.186023 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6\n
2026-03-14 09:44:27.186027 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd\n
2026-03-14 09:44:27.186031 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n
2026-03-14 09:44:27.186035 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R\n
2026-03-14 09:44:27.186040 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n
2026-03-14 09:44:27.186046 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL\n
2026-03-14 09:44:27.186050 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn\n
2026-03-14 09:44:27.186054 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To\n
2026-03-14 09:44:27.186066 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+\n
2026-03-14 09:44:27.186071 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb\n
2026-03-14 09:44:27.186075 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID\n
2026-03-14 09:44:27.186079 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r\n
2026-03-14 09:44:27.186083 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn\n
2026-03-14 09:44:27.186087 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A==\n
2026-03-14 09:44:27.186092 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.186096 | controller |       CERTIFICATE-----' ']'\n    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse
2026-03-14 09:44:27.186100 | controller |       | test step completed 1-deploy-with-tlse\n    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186105 | controller |       | starting test step 2-change-public-svc-certificate\n    logger.go:42: 09:36:49
2026-03-14 09:44:27.186109 | controller |       | watcher-tls-certs-change/2-change-public-svc-certificate | running command: [sh
2026-03-14 09:44:27.186113 | controller |       -c oc patch Certificate -n $NAMESPACE watcher-public-svc --type='json' -p='[{\"op\":
2026-03-14 09:44:27.186117 | controller |       \"replace\", \"path\": \"/spec/dnsNames\", \"value\":['watcher-public.watcher-kuttl-default.svc',
2026-03-14 09:44:27.186121 | controller |       'watcher-public.watcher-kuttl-default.svc.cluster.local']}]'\n        ]\n    logger.go:42:
2026-03-14 09:44:27.186126 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | certificate.cert-manager.io/watcher-public-svc
2026-03-14 09:44:27.186131 | controller |       patched\n    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186135 | controller |       | running command: [sh -c set -euxo pipefail\n        svc_cert=$(oc rsh -n $NAMESPACE
2026-03-14 09:44:27.186140 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n        secret_cert=$(oc
2026-03-14 09:44:27.186144 | controller |       get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.186148 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.186152 | controller |       \       if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n          exit 1\n        fi\n
2026-03-14 09:44:27.186156 | controller |       \       ]\n    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186161 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n
2026-03-14 09:44:27.186165 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186169 | controller |       | + svc_cert='-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186173 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.186177 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.186182 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186186 | controller |       | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n    logger.go:42:
2026-03-14 09:44:27.186190 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.186194 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186199 | controller |       | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n    logger.go:42:
2026-03-14 09:44:27.186203 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.186207 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186211 | controller |       | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n    logger.go:42:
2026-03-14 09:44:27.186215 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.186220 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186228 | controller |       | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n    logger.go:42:
2026-03-14 09:44:27.186233 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.186239 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186243 | controller |       | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n    logger.go:42:
2026-03-14 09:44:27.186247 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.186251 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186256 | controller |       | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n    logger.go:42:
2026-03-14 09:44:27.186260 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.186264 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186270 | controller |       | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n    logger.go:42:
2026-03-14 09:44:27.186274 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.186278 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186282 | controller |       | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n    logger.go:42:
2026-03-14 09:44:27.186287 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.186306 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186311 | controller |       | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n    logger.go:42:
2026-03-14 09:44:27.186315 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.186319 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186323 | controller |       | -----END CERTIFICATE-----'\n    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186328 | controller |       | ++ base64 --decode\n    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186332 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:44:27.186336 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186340 | controller |       | + secret_cert='-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186345 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.186349 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.186353 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186357 | controller |       | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n    logger.go:42:
2026-03-14 09:44:27.186361 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.186366 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186370 | controller |       | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n    logger.go:42:
2026-03-14 09:44:27.186374 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.186378 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186383 | controller |       | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n    logger.go:42:
2026-03-14 09:44:27.186387 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.186391 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186395 | controller |       | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n    logger.go:42:
2026-03-14 09:44:27.186403 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.186413 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186419 | controller |       | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n    logger.go:42:
2026-03-14 09:44:27.186423 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.186427 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186431 | controller |       | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n    logger.go:42:
2026-03-14 09:44:27.186436 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.186440 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186444 | controller |       | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n    logger.go:42:
2026-03-14 09:44:27.186450 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.186456 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186461 | controller |       | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n    logger.go:42:
2026-03-14 09:44:27.186467 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.186473 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186479 | controller |       | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n    logger.go:42:
2026-03-14 09:44:27.186485 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.186501 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186506 | controller |       | -----END CERTIFICATE-----'\n    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186510 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186515 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n    logger.go:42:
2026-03-14 09:44:27.186519 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n
2026-03-14 09:44:27.186523 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186527 | controller |       | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n    logger.go:42:
2026-03-14 09:44:27.186531 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n
2026-03-14 09:44:27.186535 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186540 | controller |       | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n    logger.go:42:
2026-03-14 09:44:27.186544 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n
2026-03-14 09:44:27.186548 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186552 | controller |       | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n    logger.go:42:
2026-03-14 09:44:27.186556 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n
2026-03-14 09:44:27.186561 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186565 | controller |       | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n    logger.go:42:
2026-03-14 09:44:27.186571 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n
2026-03-14 09:44:27.186575 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186580 | controller |       | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n    logger.go:42:
2026-03-14 09:44:27.186584 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n
2026-03-14 09:44:27.186588 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186592 | controller |       | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n    logger.go:42:
2026-03-14 09:44:27.186605 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n
2026-03-14 09:44:27.186611 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186617 | controller |       | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n    logger.go:42:
2026-03-14 09:44:27.186623 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n
2026-03-14 09:44:27.186700 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186712 | controller |       | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n    logger.go:42:
2026-03-14 09:44:27.186719 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n
2026-03-14 09:44:27.186725 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186731 | controller |       | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n    logger.go:42:
2026-03-14 09:44:27.186737 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n
2026-03-14 09:44:27.186743 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186749 | controller |       | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42:
2026-03-14 09:44:27.186755 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw\n
2026-03-14 09:44:27.186761 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186766 | controller |       | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz\n    logger.go:42:
2026-03-14 09:44:27.186771 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN\n
2026-03-14 09:44:27.186775 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186779 | controller |       | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz\n    logger.go:42:
2026-03-14 09:44:27.186783 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn\n
2026-03-14 09:44:27.186788 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186792 | controller |       | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk\n    logger.go:42:
2026-03-14 09:44:27.186797 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou\n
2026-03-14 09:44:27.186803 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186809 | controller |       | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a\n    logger.go:42:
2026-03-14 09:44:27.186818 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM\n
2026-03-14 09:44:27.186825 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186834 | controller |       | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7\n    logger.go:42:
2026-03-14 09:44:27.186840 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl\n
2026-03-14 09:44:27.186846 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186852 | controller |       | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR\n    logger.go:42:
2026-03-14 09:44:27.186859 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG\n
2026-03-14 09:44:27.186865 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186871 | controller |       | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J\n    logger.go:42:
2026-03-14 09:44:27.186877 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX\n
2026-03-14 09:44:27.186883 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186887 | controller |       | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z\n    logger.go:42:
2026-03-14 09:44:27.186892 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc\n
2026-03-14 09:44:27.186898 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186913 | controller |       | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG\n    logger.go:42:
2026-03-14 09:44:27.186920 | controller |       09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK\n
2026-03-14 09:44:27.186927 | controller |       \   logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186933 | controller |       | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A\n    logger.go:42: 09:36:49
2026-03-14 09:44:27.186939 | controller |       | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----'
2026-03-14 09:44:27.186945 | controller |       ']'\n    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.186950 | controller |       | test step completed 2-change-public-svc-certificate\n    logger.go:42: 09:36:49
2026-03-14 09:44:27.186956 | controller |       | watcher-tls-certs-change/3-change-internal-svc-certificate | starting test step
2026-03-14 09:44:27.186962 | controller |       3-change-internal-svc-certificate\n    logger.go:42: 09:36:49 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.186968 | controller |       | running command: [sh -c oc patch Certificate -n $NAMESPACE watcher-internal-svc
2026-03-14 09:44:27.186974 | controller |       --type='json' -p='[{\"op\": \"replace\", \"path\": \"/spec/dnsNames\", \"value\":['watcher-internal.watcher-kuttl-default.svc',
2026-03-14 09:44:27.186979 | controller |       'watcher-internal.watcher-kuttl-default.svc.cluster.local']}]'\n        ]\n    logger.go:42:
2026-03-14 09:44:27.186985 | controller |       09:36:49 | watcher-tls-certs-change/3-change-internal-svc-certificate | certificate.cert-manager.io/watcher-internal-svc
2026-03-14 09:44:27.186991 | controller |       patched\n    logger.go:42: 09:36:49 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.186997 | controller |       | running command: [sh -c set -euxo pipefail\n        svc_cert=$(oc rsh -n $NAMESPACE
2026-03-14 09:44:27.187003 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        secret_cert=$(oc
2026-03-14 09:44:27.187011 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.187017 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.187021 | controller |       \       if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n          exit 1\n        fi\n
2026-03-14 09:44:27.187026 | controller |       \       ]\n    logger.go:42: 09:36:49 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187030 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:44:27.187034 | controller |       \   logger.go:42: 09:36:50 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187039 | controller |       | error: Internal error occurred: error executing command in container: container
2026-03-14 09:44:27.187043 | controller |       is not created or running\n    logger.go:42: 09:36:50 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187047 | controller |       | + svc_cert=\n    logger.go:42: 09:36:51 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187052 | controller |       | running command: [sh -c set -euxo pipefail\n        svc_cert=$(oc rsh -n $NAMESPACE
2026-03-14 09:44:27.187058 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        secret_cert=$(oc
2026-03-14 09:44:27.187063 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.187068 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.187072 | controller |       \       if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n          exit 1\n        fi\n
2026-03-14 09:44:27.187076 | controller |       \       ]\n    logger.go:42: 09:36:51 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187081 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:44:27.187085 | controller |       \   logger.go:42: 09:36:51 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187089 | controller |       | error: unable to upgrade connection: container not found (\"watcher-api\")\n    logger.go:42:
2026-03-14 09:44:27.187093 | controller |       09:36:51 | watcher-tls-certs-change/3-change-internal-svc-certificate | + svc_cert=\n
2026-03-14 09:44:27.187098 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187102 | controller |       | running command: [sh -c set -euxo pipefail\n        svc_cert=$(oc rsh -n $NAMESPACE
2026-03-14 09:44:27.187106 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n        secret_cert=$(oc
2026-03-14 09:44:27.187110 | controller |       get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}'
2026-03-14 09:44:27.187115 | controller |       | base64 --decode)\n        # ensure that the svc secret and cert secret match\n
2026-03-14 09:44:27.187119 | controller |       \       if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n          exit 1\n        fi\n
2026-03-14 09:44:27.187129 | controller |       \       ]\n    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187135 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n
2026-03-14 09:44:27.187139 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187144 | controller |       | + svc_cert='-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187149 | controller |       | MIID8DCCAligAwIBAgIQajyja/okveMym9ZKBf72LjANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:44:27.187155 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDkzNjUwWhcNMzEw\n
2026-03-14 09:44:27.187166 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187173 | controller |       | MzEzMDkzNjUwWjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4S4p\n    logger.go:42:
2026-03-14 09:44:27.187179 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | yP8aQfvOUpKYA2csCMYtnRCvHUQjFn5L/0uId8KEre8xVSzRZbeTV05hkxlklYaZ\n
2026-03-14 09:44:27.187185 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187222 | controller |       | zh79ln9aDCOT6kanAVAppwiA7Gmyw3cLC5SnLT1nAHipvF3r7DqzI7MFgO5gRFgW\n    logger.go:42:
2026-03-14 09:44:27.187228 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | HpAoSIsnBPwXt4hnmrw2Vtzdw34ig+letfnOxuZWaU4h8krF+8hCGJ/RLq99ajJy\n
2026-03-14 09:44:27.187233 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187237 | controller |       | G6CFAwh3YjWCoqeemDvIW92p2Up9uTyT7Bxzmbe+ybEFSXQAUYzQde3B3mWIBLnz\n    logger.go:42:
2026-03-14 09:44:27.187242 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | esOz/0G3/3e24Aj9+7zpcbd0pVuVMUXCV9pEcPo1kTKX+KBgvS6Z+oi7lkeHPXYw\n
2026-03-14 09:44:27.187246 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187250 | controller |       | O/lURQupvVThqDbWDwIDAQABo4HLMIHIMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n    logger.go:42:
2026-03-14 09:44:27.187255 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFEeYb+Q03hEB\n
2026-03-14 09:44:27.187260 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187266 | controller |       | dG616pECdMfMDrE8MHIGA1UdEQEB/wRoMGaCKndhdGNoZXItaW50ZXJuYWwud2F0\n    logger.go:42:
2026-03-14 09:44:27.187272 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | Y2hlci1rdXR0bC1kZWZhdWx0LnN2Y4I4d2F0Y2hlci1pbnRlcm5hbC53YXRjaGVy\n
2026-03-14 09:44:27.187279 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187285 | controller |       | LWt1dHRsLWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQAD\n    logger.go:42:
2026-03-14 09:44:27.187313 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | ggGBAK6NF533YWWeQT84GLLzJ0ogHXKIDKZVRwqLaNXLWGG0W03Cow9vch6+Fv0b\n
2026-03-14 09:44:27.187318 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187323 | controller |       | rkJisw8FbyD772PLovRl0hsUOW2DQw+R7qtNsCHUTxLMDnBkJ1QBYPJS80LumsGR\n    logger.go:42:
2026-03-14 09:44:27.187327 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | vVXFLl69tqGXkoaOyYgogD7HHJO69quA7qENirYSK1RHdmoqF5CX2qKJnLdnEgmP\n
2026-03-14 09:44:27.187332 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187337 | controller |       | ZFGE1JshS1XaVoz4DYDTlqq8FHZ0/Jigrj8EG1Rjpv6+qA8m2ZEsk8ISyFmzwWqi\n    logger.go:42:
2026-03-14 09:44:27.187343 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | FGw1GV874NeqiB3ip4atDzH2ossazs4WNn65dA757uk6lICb2vKbXzNTadZRTQSM\n
2026-03-14 09:44:27.187349 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187355 | controller |       | FehTsYpmDx5lc1y9K1MpBOmZmGRXIfHkrg2YVRsFq72bZQlRfop1dOgXDKRImakm\n    logger.go:42:
2026-03-14 09:44:27.187361 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | F9+dzFNDrHO6H3MHI6bnC2DQUWI2FOvAJ8L93wedHbyR8sJ8ObnJli7pngCIY1Rp\n
2026-03-14 09:44:27.187367 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187373 | controller |       | E8tkXhzRn5kiy7dXmeP4LYghoCN/8q0zIct6CNUFL3VMCCm9EWruWCiFr+G60Nz0\n    logger.go:42:
2026-03-14 09:44:27.187379 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | G8kFPA==\n
2026-03-14 09:44:27.187389 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187395 | controller |       | -----END CERTIFICATE-----'\n    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187409 | controller |       | ++ base64 --decode\n    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187416 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n
2026-03-14 09:44:27.187422 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187428 | controller |       | + secret_cert='-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187434 | controller |       | MIID8DCCAligAwIBAgIQajyja/okveMym9ZKBf72LjANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:44:27.187439 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDkzNjUwWhcNMzEw\n
2026-03-14 09:44:27.187444 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187448 | controller |       | MzEzMDkzNjUwWjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4S4p\n    logger.go:42:
2026-03-14 09:44:27.187454 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | yP8aQfvOUpKYA2csCMYtnRCvHUQjFn5L/0uId8KEre8xVSzRZbeTV05hkxlklYaZ\n
2026-03-14 09:44:27.187463 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187470 | controller |       | zh79ln9aDCOT6kanAVAppwiA7Gmyw3cLC5SnLT1nAHipvF3r7DqzI7MFgO5gRFgW\n    logger.go:42:
2026-03-14 09:44:27.187476 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | HpAoSIsnBPwXt4hnmrw2Vtzdw34ig+letfnOxuZWaU4h8krF+8hCGJ/RLq99ajJy\n
2026-03-14 09:44:27.187482 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187488 | controller |       | G6CFAwh3YjWCoqeemDvIW92p2Up9uTyT7Bxzmbe+ybEFSXQAUYzQde3B3mWIBLnz\n    logger.go:42:
2026-03-14 09:44:27.187494 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | esOz/0G3/3e24Aj9+7zpcbd0pVuVMUXCV9pEcPo1kTKX+KBgvS6Z+oi7lkeHPXYw\n
2026-03-14 09:44:27.187500 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187508 | controller |       | O/lURQupvVThqDbWDwIDAQABo4HLMIHIMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n    logger.go:42:
2026-03-14 09:44:27.187514 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFEeYb+Q03hEB\n
2026-03-14 09:44:27.187520 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187525 | controller |       | dG616pECdMfMDrE8MHIGA1UdEQEB/wRoMGaCKndhdGNoZXItaW50ZXJuYWwud2F0\n    logger.go:42:
2026-03-14 09:44:27.187531 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | Y2hlci1rdXR0bC1kZWZhdWx0LnN2Y4I4d2F0Y2hlci1pbnRlcm5hbC53YXRjaGVy\n
2026-03-14 09:44:27.187566 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187571 | controller |       | LWt1dHRsLWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQAD\n    logger.go:42:
2026-03-14 09:44:27.187576 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | ggGBAK6NF533YWWeQT84GLLzJ0ogHXKIDKZVRwqLaNXLWGG0W03Cow9vch6+Fv0b\n
2026-03-14 09:44:27.187580 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187584 | controller |       | rkJisw8FbyD772PLovRl0hsUOW2DQw+R7qtNsCHUTxLMDnBkJ1QBYPJS80LumsGR\n    logger.go:42:
2026-03-14 09:44:27.187588 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | vVXFLl69tqGXkoaOyYgogD7HHJO69quA7qENirYSK1RHdmoqF5CX2qKJnLdnEgmP\n
2026-03-14 09:44:27.187595 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187599 | controller |       | ZFGE1JshS1XaVoz4DYDTlqq8FHZ0/Jigrj8EG1Rjpv6+qA8m2ZEsk8ISyFmzwWqi\n    logger.go:42:
2026-03-14 09:44:27.187604 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | FGw1GV874NeqiB3ip4atDzH2ossazs4WNn65dA757uk6lICb2vKbXzNTadZRTQSM\n
2026-03-14 09:44:27.187608 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187612 | controller |       | FehTsYpmDx5lc1y9K1MpBOmZmGRXIfHkrg2YVRsFq72bZQlRfop1dOgXDKRImakm\n    logger.go:42:
2026-03-14 09:44:27.187616 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | F9+dzFNDrHO6H3MHI6bnC2DQUWI2FOvAJ8L93wedHbyR8sJ8ObnJli7pngCIY1Rp\n
2026-03-14 09:44:27.187620 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187625 | controller |       | E8tkXhzRn5kiy7dXmeP4LYghoCN/8q0zIct6CNUFL3VMCCm9EWruWCiFr+G60Nz0\n    logger.go:42:
2026-03-14 09:44:27.187629 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | G8kFPA==\n
2026-03-14 09:44:27.187633 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187645 | controller |       | -----END CERTIFICATE-----'\n    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187649 | controller |       | + '[' '-----BEGIN CERTIFICATE-----\n    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187654 | controller |       | MIID8DCCAligAwIBAgIQajyja/okveMym9ZKBf72LjANBgkqhkiG9w0BAQwFADAa\n    logger.go:42:
2026-03-14 09:44:27.187658 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDkzNjUwWhcNMzEw\n
2026-03-14 09:44:27.187662 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187667 | controller |       | MzEzMDkzNjUwWjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4S4p\n    logger.go:42:
2026-03-14 09:44:27.187671 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | yP8aQfvOUpKYA2csCMYtnRCvHUQjFn5L/0uId8KEre8xVSzRZbeTV05hkxlklYaZ\n
2026-03-14 09:44:27.187675 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187679 | controller |       | zh79ln9aDCOT6kanAVAppwiA7Gmyw3cLC5SnLT1nAHipvF3r7DqzI7MFgO5gRFgW\n    logger.go:42:
2026-03-14 09:44:27.187683 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | HpAoSIsnBPwXt4hnmrw2Vtzdw34ig+letfnOxuZWaU4h8krF+8hCGJ/RLq99ajJy\n
2026-03-14 09:44:27.187688 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187692 | controller |       | G6CFAwh3YjWCoqeemDvIW92p2Up9uTyT7Bxzmbe+ybEFSXQAUYzQde3B3mWIBLnz\n    logger.go:42:
2026-03-14 09:44:27.187696 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | esOz/0G3/3e24Aj9+7zpcbd0pVuVMUXCV9pEcPo1kTKX+KBgvS6Z+oi7lkeHPXYw\n
2026-03-14 09:44:27.187700 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187705 | controller |       | O/lURQupvVThqDbWDwIDAQABo4HLMIHIMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n    logger.go:42:
2026-03-14 09:44:27.187711 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFEeYb+Q03hEB\n
2026-03-14 09:44:27.187717 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187723 | controller |       | dG616pECdMfMDrE8MHIGA1UdEQEB/wRoMGaCKndhdGNoZXItaW50ZXJuYWwud2F0\n    logger.go:42:
2026-03-14 09:44:27.187729 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | Y2hlci1rdXR0bC1kZWZhdWx0LnN2Y4I4d2F0Y2hlci1pbnRlcm5hbC53YXRjaGVy\n
2026-03-14 09:44:27.187735 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187745 | controller |       | LWt1dHRsLWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQAD\n    logger.go:42:
2026-03-14 09:44:27.187751 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | ggGBAK6NF533YWWeQT84GLLzJ0ogHXKIDKZVRwqLaNXLWGG0W03Cow9vch6+Fv0b\n
2026-03-14 09:44:27.187755 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187759 | controller |       | rkJisw8FbyD772PLovRl0hsUOW2DQw+R7qtNsCHUTxLMDnBkJ1QBYPJS80LumsGR\n    logger.go:42:
2026-03-14 09:44:27.187763 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | vVXFLl69tqGXkoaOyYgogD7HHJO69quA7qENirYSK1RHdmoqF5CX2qKJnLdnEgmP\n
2026-03-14 09:44:27.187768 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187774 | controller |       | ZFGE1JshS1XaVoz4DYDTlqq8FHZ0/Jigrj8EG1Rjpv6+qA8m2ZEsk8ISyFmzwWqi\n    logger.go:42:
2026-03-14 09:44:27.187779 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | FGw1GV874NeqiB3ip4atDzH2ossazs4WNn65dA757uk6lICb2vKbXzNTadZRTQSM\n
2026-03-14 09:44:27.187783 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187787 | controller |       | FehTsYpmDx5lc1y9K1MpBOmZmGRXIfHkrg2YVRsFq72bZQlRfop1dOgXDKRImakm\n    logger.go:42:
2026-03-14 09:44:27.187791 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | F9+dzFNDrHO6H3MHI6bnC2DQUWI2FOvAJ8L93wedHbyR8sJ8ObnJli7pngCIY1Rp\n
2026-03-14 09:44:27.187796 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187800 | controller |       | E8tkXhzRn5kiy7dXmeP4LYghoCN/8q0zIct6CNUFL3VMCCm9EWruWCiFr+G60Nz0\n    logger.go:42:
2026-03-14 09:44:27.187804 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | G8kFPA==\n
2026-03-14 09:44:27.187808 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187812 | controller |       | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n    logger.go:42:
2026-03-14 09:44:27.187817 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | MIID8DCCAligAwIBAgIQajyja/okveMym9ZKBf72LjANBgkqhkiG9w0BAQwFADAa\n
2026-03-14 09:44:27.187828 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187833 | controller |       | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDkzNjUwWhcNMzEw\n    logger.go:42:
2026-03-14 09:44:27.187837 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | MzEzMDkzNjUwWjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4S4p\n
2026-03-14 09:44:27.187841 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187845 | controller |       | yP8aQfvOUpKYA2csCMYtnRCvHUQjFn5L/0uId8KEre8xVSzRZbeTV05hkxlklYaZ\n    logger.go:42:
2026-03-14 09:44:27.187850 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | zh79ln9aDCOT6kanAVAppwiA7Gmyw3cLC5SnLT1nAHipvF3r7DqzI7MFgO5gRFgW\n
2026-03-14 09:44:27.187856 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187860 | controller |       | HpAoSIsnBPwXt4hnmrw2Vtzdw34ig+letfnOxuZWaU4h8krF+8hCGJ/RLq99ajJy\n    logger.go:42:
2026-03-14 09:44:27.187864 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | G6CFAwh3YjWCoqeemDvIW92p2Up9uTyT7Bxzmbe+ybEFSXQAUYzQde3B3mWIBLnz\n
2026-03-14 09:44:27.187869 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187873 | controller |       | esOz/0G3/3e24Aj9+7zpcbd0pVuVMUXCV9pEcPo1kTKX+KBgvS6Z+oi7lkeHPXYw\n    logger.go:42:
2026-03-14 09:44:27.187877 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | O/lURQupvVThqDbWDwIDAQABo4HLMIHIMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\n
2026-03-14 09:44:27.187881 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187887 | controller |       | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFEeYb+Q03hEB\n    logger.go:42:
2026-03-14 09:44:27.187891 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | dG616pECdMfMDrE8MHIGA1UdEQEB/wRoMGaCKndhdGNoZXItaW50ZXJuYWwud2F0\n
2026-03-14 09:44:27.187896 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187900 | controller |       | Y2hlci1rdXR0bC1kZWZhdWx0LnN2Y4I4d2F0Y2hlci1pbnRlcm5hbC53YXRjaGVy\n    logger.go:42:
2026-03-14 09:44:27.187907 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | LWt1dHRsLWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQAD\n
2026-03-14 09:44:27.187912 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187918 | controller |       | ggGBAK6NF533YWWeQT84GLLzJ0ogHXKIDKZVRwqLaNXLWGG0W03Cow9vch6+Fv0b\n    logger.go:42:
2026-03-14 09:44:27.187922 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | rkJisw8FbyD772PLovRl0hsUOW2DQw+R7qtNsCHUTxLMDnBkJ1QBYPJS80LumsGR\n
2026-03-14 09:44:27.187927 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187932 | controller |       | vVXFLl69tqGXkoaOyYgogD7HHJO69quA7qENirYSK1RHdmoqF5CX2qKJnLdnEgmP\n    logger.go:42:
2026-03-14 09:44:27.187937 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | ZFGE1JshS1XaVoz4DYDTlqq8FHZ0/Jigrj8EG1Rjpv6+qA8m2ZEsk8ISyFmzwWqi\n
2026-03-14 09:44:27.187941 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187945 | controller |       | FGw1GV874NeqiB3ip4atDzH2ossazs4WNn65dA757uk6lICb2vKbXzNTadZRTQSM\n    logger.go:42:
2026-03-14 09:44:27.187950 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | FehTsYpmDx5lc1y9K1MpBOmZmGRXIfHkrg2YVRsFq72bZQlRfop1dOgXDKRImakm\n
2026-03-14 09:44:27.187954 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187958 | controller |       | F9+dzFNDrHO6H3MHI6bnC2DQUWI2FOvAJ8L93wedHbyR8sJ8ObnJli7pngCIY1Rp\n    logger.go:42:
2026-03-14 09:44:27.187962 | controller |       09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate | E8tkXhzRn5kiy7dXmeP4LYghoCN/8q0zIct6CNUFL3VMCCm9EWruWCiFr+G60Nz0\n
2026-03-14 09:44:27.187967 | controller |       \   logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187971 | controller |       | G8kFPA==\n    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187975 | controller |       | -----END CERTIFICATE-----' ']'\n    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.187979 | controller |       | test step completed 3-change-internal-svc-certificate\n    logger.go:42: 09:36:52
2026-03-14 09:44:27.187983 | controller |       | watcher-tls-certs-change/4-cleanup-watcher | starting test step 4-cleanup-watcher\n
2026-03-14 09:44:27.187989 | controller |       \   logger.go:42: 09:37:00 | watcher-tls-certs-change/4-cleanup-watcher | test step
2026-03-14 09:44:27.187994 | controller |       completed 4-cleanup-watcher\n    logger.go:42: 09:37:00 | watcher-tls-certs-change/5-clenaup-certs
2026-03-14 09:44:27.187998 | controller |       | starting test step 5-clenaup-certs\n    logger.go:42: 09:37:00 | watcher-tls-certs-change/5-clenaup-certs
2026-03-14 09:44:27.188007 | controller |       | test step completed 5-clenaup-certs\n    logger.go:42: 09:37:00 | watcher-tls-certs-change
2026-03-14 09:44:27.188012 | controller |       | skipping kubernetes event logging\n=== CONT  kuttl/harness/watcher-tls\n    logger.go:42:
2026-03-14 09:44:27.188016 | controller |       09:37:00 | watcher-tls | Skipping creation of user-supplied namespace: watcher-kuttl-default\n
2026-03-14 09:44:27.188021 | controller |       \   logger.go:42: 09:37:00 | watcher-tls/0-cleanup-watcher | starting test step
2026-03-14 09:44:27.188025 | controller |       0-cleanup-watcher\n    logger.go:42: 09:37:00 | watcher-tls/0-cleanup-watcher |
2026-03-14 09:44:27.188032 | controller |       test step completed 0-cleanup-watcher\n    logger.go:42: 09:37:00 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188036 | controller |       | starting test step 1-deploy-with-tlse\n    logger.go:42: 09:37:00 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188041 | controller |       | Certificate:watcher-kuttl-default/watcher-internal-svc created\n    logger.go:42:
2026-03-14 09:44:27.188045 | controller |       09:37:00 | watcher-tls/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc
2026-03-14 09:44:27.188049 | controller |       created\n    logger.go:42: 09:37:00 | watcher-tls/1-deploy-with-tlse | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:44:27.188053 | controller |       created\n    logger.go:42: 09:37:00 | watcher-tls/1-deploy-with-tlse | running command:
2026-03-14 09:44:27.188058 | controller |       [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient -- openstack
2026-03-14 09:44:27.188062 | controller |       service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc
2026-03-14 09:44:27.188066 | controller |       exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.188070 | controller |       -c Type -c ID | grep watcher| awk '{print $1}')\n        [ $(oc get -n ${NAMESPACE}
2026-03-14 09:44:27.188074 | controller |       keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [
2026-03-14 09:44:27.188079 | controller |       -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\"
2026-03-14 09:44:27.188083 | controller |       ]\n        [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o
2026-03-14 09:44:27.188087 | controller |       jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n        [ \"$(oc
2026-03-14 09:44:27.188091 | controller |       get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64
2026-03-14 09:44:27.188096 | controller |       -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2
2026-03-14 09:44:27.188102 | controller |       ]\n        # check that both endpoints have https set\n        oc exec -n ${NAMESPACE}
2026-03-14 09:44:27.188108 | controller |       openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https)
2026-03-14 09:44:27.188114 | controller |       == 2 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:44:27.188120 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:44:27.188126 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:44:27.188132 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:44:27.188138 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:44:27.188144 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:44:27.188148 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:44:27.188152 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:44:27.188156 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:44:27.188160 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:37:00 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188165 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.188169 | controller |       value -c Name -c Type\n    logger.go:42: 09:37:00 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188173 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:37:03 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188177 | controller |       | + '[' 0 == 1 ']'\n    logger.go:42: 09:37:04 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188181 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:44:27.188188 | controller |       -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:44:27.188192 | controller |       \       SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.188196 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n        [
2026-03-14 09:44:27.188200 | controller |       $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.188205 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl
2026-03-14 09:44:27.188218 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        [ \"$(oc get -n ${NAMESPACE} secret
2026-03-14 09:44:27.188223 | controller |       watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\"
2026-03-14 09:44:27.188227 | controller |       == 1 ]\n        [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.188231 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\"
2026-03-14 09:44:27.188235 | controller |       == 2 ]\n        # check that both endpoints have https set\n        oc exec -n ${NAMESPACE}
2026-03-14 09:44:27.188240 | controller |       openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https)
2026-03-14 09:44:27.188244 | controller |       == 2 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:44:27.188248 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:44:27.188252 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:44:27.188256 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:44:27.188260 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:44:27.188265 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:44:27.188269 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:44:27.188273 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:44:27.188277 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:44:27.188281 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:37:04 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188286 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.188305 | controller |       value -c Name -c Type\n    logger.go:42: 09:37:04 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188313 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:37:06 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188318 | controller |       | + '[' 0 == 1 ']'\n    logger.go:42: 09:37:08 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188322 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:44:27.188326 | controller |       -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:44:27.188330 | controller |       \       SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.188335 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n        [
2026-03-14 09:44:27.188339 | controller |       $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.188343 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl
2026-03-14 09:44:27.188347 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        [ \"$(oc get -n ${NAMESPACE} secret
2026-03-14 09:44:27.188351 | controller |       watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\"
2026-03-14 09:44:27.188355 | controller |       == 1 ]\n        [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.188362 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\"
2026-03-14 09:44:27.188366 | controller |       == 2 ]\n        # check that both endpoints have https set\n        oc exec -n ${NAMESPACE}
2026-03-14 09:44:27.188370 | controller |       openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https)
2026-03-14 09:44:27.188376 | controller |       == 2 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:44:27.188382 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:44:27.188388 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:44:27.188393 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:44:27.188399 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:44:27.188405 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:44:27.188411 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:44:27.188417 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:44:27.188423 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:44:27.188429 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:37:08 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188435 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.188441 | controller |       value -c Name -c Type\n    logger.go:42: 09:37:08 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188456 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:37:10 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188464 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:37:10 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188470 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.188477 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:37:10 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188487 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:37:10 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188493 | controller |       | ++ grep watcher\n    logger.go:42: 09:37:12 | watcher-tls/1-deploy-with-tlse |
2026-03-14 09:44:27.188498 | controller |       + SERVICEID=e0a908c724554093bb8de86515131a2c\n    logger.go:42: 09:37:12 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188503 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.188507 | controller |       \   logger.go:42: 09:37:12 | watcher-tls/1-deploy-with-tlse | + '[' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.188511 | controller |       == e0a908c724554093bb8de86515131a2c ']'\n    logger.go:42: 09:37:12 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188515 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:44:27.188519 | controller |       \   logger.go:42: 09:37:12 | watcher-tls/1-deploy-with-tlse | + '[' -n '' ']'\n
2026-03-14 09:44:27.188524 | controller |       \   logger.go:42: 09:37:13 | watcher-tls/1-deploy-with-tlse | running command: [sh
2026-03-14 09:44:27.188528 | controller |       -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient -- openstack
2026-03-14 09:44:27.188534 | controller |       service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc
2026-03-14 09:44:27.188538 | controller |       exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.188542 | controller |       -c Type -c ID | grep watcher| awk '{print $1}')\n        [ $(oc get -n ${NAMESPACE}
2026-03-14 09:44:27.188546 | controller |       keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [
2026-03-14 09:44:27.188552 | controller |       -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\"
2026-03-14 09:44:27.188557 | controller |       ]\n        [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o
2026-03-14 09:44:27.188561 | controller |       jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n        [ \"$(oc
2026-03-14 09:44:27.188565 | controller |       get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64
2026-03-14 09:44:27.188569 | controller |       -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2
2026-03-14 09:44:27.188574 | controller |       ]\n        # check that both endpoints have https set\n        oc exec -n ${NAMESPACE}
2026-03-14 09:44:27.188578 | controller |       openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https)
2026-03-14 09:44:27.188582 | controller |       == 2 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:44:27.188586 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:44:27.188590 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:44:27.188595 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:44:27.188600 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:44:27.188605 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:44:27.188611 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:44:27.188617 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:44:27.188623 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:44:27.188629 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:37:13 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188634 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.188640 | controller |       value -c Name -c Type\n    logger.go:42: 09:37:13 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188646 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:37:15 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188652 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:37:15 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188658 | controller |       | ++ grep watcher\n    logger.go:42: 09:37:15 | watcher-tls/1-deploy-with-tlse |
2026-03-14 09:44:27.188663 | controller |       ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.188669 | controller |       value -c Name -c Type -c ID\n    logger.go:42: 09:37:15 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188675 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:37:17 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188690 | controller |       | + SERVICEID=e0a908c724554093bb8de86515131a2c\n    logger.go:42: 09:37:17 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188697 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.188704 | controller |       \   logger.go:42: 09:37:17 | watcher-tls/1-deploy-with-tlse | + '[' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.188710 | controller |       == e0a908c724554093bb8de86515131a2c ']'\n    logger.go:42: 09:37:17 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188716 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:44:27.188722 | controller |       \   logger.go:42: 09:37:18 | watcher-tls/1-deploy-with-tlse | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.188728 | controller |       ']'\n    logger.go:42: 09:37:18 | watcher-tls/1-deploy-with-tlse | ++ oc get -n
2026-03-14 09:44:27.188736 | controller |       watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n
2026-03-14 09:44:27.188741 | controller |       \   logger.go:42: 09:37:18 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1\n
2026-03-14 09:44:27.188746 | controller |       \   logger.go:42: 09:37:18 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n    logger.go:42:
2026-03-14 09:44:27.188750 | controller |       09:37:18 | watcher-tls/1-deploy-with-tlse | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:44:27.188758 | controller |       09:37:18 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret
2026-03-14 09:44:27.188762 | controller |       watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42:
2026-03-14 09:44:27.188766 | controller |       09:37:18 | watcher-tls/1-deploy-with-tlse | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n
2026-03-14 09:44:27.188771 | controller |       \   logger.go:42: 09:37:18 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n    logger.go:42:
2026-03-14 09:44:27.188775 | controller |       09:37:18 | watcher-tls/1-deploy-with-tlse | + '[' 2 == 2 ']'\n    logger.go:42:
2026-03-14 09:44:27.188779 | controller |       09:37:18 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.188783 | controller |       -- openstack endpoint list\n    logger.go:42: 09:37:18 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188788 | controller |       | + grep infra-optim\n    logger.go:42: 09:37:18 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188792 | controller |       | ++ grep -c https\n    logger.go:42: 09:37:20 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188796 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:37:21 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188800 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:44:27.188804 | controller |       -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:44:27.188809 | controller |       \       SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.188813 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n        [
2026-03-14 09:44:27.188817 | controller |       $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.188821 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl
2026-03-14 09:44:27.188825 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        [ \"$(oc get -n ${NAMESPACE} secret
2026-03-14 09:44:27.188829 | controller |       watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\"
2026-03-14 09:44:27.188833 | controller |       == 1 ]\n        [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.188838 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\"
2026-03-14 09:44:27.188842 | controller |       == 2 ]\n        # check that both endpoints have https set\n        oc exec -n ${NAMESPACE}
2026-03-14 09:44:27.188846 | controller |       openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https)
2026-03-14 09:44:27.188850 | controller |       == 2 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:44:27.188855 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:44:27.188859 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:44:27.188863 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:44:27.188867 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:44:27.188871 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:44:27.188875 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:44:27.188881 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:44:27.188885 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:44:27.188896 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:37:21 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188901 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.188905 | controller |       value -c Name -c Type\n    logger.go:42: 09:37:21 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188911 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:37:23 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188916 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:37:23 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188922 | controller |       | ++ grep watcher\n    logger.go:42: 09:37:23 | watcher-tls/1-deploy-with-tlse |
2026-03-14 09:44:27.188928 | controller |       ++ awk '{print $1}'\n    logger.go:42: 09:37:23 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188934 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.188940 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:37:25 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188947 | controller |       | + SERVICEID=e0a908c724554093bb8de86515131a2c\n    logger.go:42: 09:37:25 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188953 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.188959 | controller |       \   logger.go:42: 09:37:25 | watcher-tls/1-deploy-with-tlse | + '[' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.188965 | controller |       == e0a908c724554093bb8de86515131a2c ']'\n    logger.go:42: 09:37:25 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.188971 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:44:27.188980 | controller |       \   logger.go:42: 09:37:25 | watcher-tls/1-deploy-with-tlse | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.188986 | controller |       ']'\n    logger.go:42: 09:37:25 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n
2026-03-14 09:44:27.188992 | controller |       \   logger.go:42: 09:37:25 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.188998 | controller |       secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n    logger.go:42:
2026-03-14 09:44:27.189004 | controller |       09:37:25 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1\n    logger.go:42:
2026-03-14 09:44:27.189012 | controller |       09:37:26 | watcher-tls/1-deploy-with-tlse | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:44:27.189018 | controller |       09:37:26 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n    logger.go:42: 09:37:26
2026-03-14 09:44:27.189022 | controller |       | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.189027 | controller |       -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42: 09:37:26 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.189035 | controller |       | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n    logger.go:42:
2026-03-14 09:44:27.189040 | controller |       09:37:26 | watcher-tls/1-deploy-with-tlse | + '[' 2 == 2 ']'\n    logger.go:42:
2026-03-14 09:44:27.189044 | controller |       09:37:26 | watcher-tls/1-deploy-with-tlse | + grep infra-optim\n    logger.go:42:
2026-03-14 09:44:27.189048 | controller |       09:37:26 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.189053 | controller |       -- openstack endpoint list\n    logger.go:42: 09:37:26 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.189057 | controller |       | ++ grep -c https\n    logger.go:42: 09:37:28 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.189061 | controller |       | + '[' 2 == 2 ']'\n    logger.go:42: 09:37:28 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.189067 | controller |       | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:44:27.189072 | controller |       \   logger.go:42: 09:37:28 | watcher-tls/1-deploy-with-tlse | + '[' '' == '' ']'\n
2026-03-14 09:44:27.189076 | controller |       \   logger.go:42: 09:37:28 | watcher-tls/1-deploy-with-tlse | + exit 0\n    logger.go:42:
2026-03-14 09:44:27.189081 | controller |       09:37:29 | watcher-tls/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:44:27.189087 | controller |       \       oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.189093 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n ${NAMESPACE}
2026-03-14 09:44:27.189099 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep
2026-03-14 09:44:27.189105 | controller |       watcher| awk '{print $1}')\n        [ $(oc get -n ${NAMESPACE} keystoneservice watcher
2026-03-14 09:44:27.189111 | controller |       -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE}
2026-03-14 09:44:27.189117 | controller |       watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n        [ \"$(oc get
2026-03-14 09:44:27.189123 | controller |       -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64
2026-03-14 09:44:27.189129 | controller |       -d|grep -c 'ssl=1')\" == 1 ]\n        [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.189144 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\"
2026-03-14 09:44:27.189151 | controller |       == 2 ]\n        # check that both endpoints have https set\n        oc exec -n ${NAMESPACE}
2026-03-14 09:44:27.189157 | controller |       openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https)
2026-03-14 09:44:27.189162 | controller |       == 2 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:44:27.189168 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:44:27.189174 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:44:27.189181 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:44:27.189187 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:44:27.189193 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:44:27.189199 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:44:27.189206 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:44:27.189212 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:44:27.189218 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:37:29 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.189224 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.189231 | controller |       value -c Name -c Type\n    logger.go:42: 09:37:29 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.189237 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:37:31 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.189243 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:37:31 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.189250 | controller |       | ++ grep watcher\n    logger.go:42: 09:37:31 | watcher-tls/1-deploy-with-tlse |
2026-03-14 09:44:27.189256 | controller |       ++ awk '{print $1}'\n    logger.go:42: 09:37:31 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.189262 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.189268 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.189278 | controller |       | + SERVICEID=e0a908c724554093bb8de86515131a2c\n    logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.189285 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.189314 | controller |       \   logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse | + '[' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.189321 | controller |       == e0a908c724554093bb8de86515131a2c ']'\n    logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.189327 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:44:27.189334 | controller |       \   logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.189340 | controller |       ']'\n    logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n
2026-03-14 09:44:27.189347 | controller |       \   logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.189353 | controller |       secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n    logger.go:42:
2026-03-14 09:44:27.189359 | controller |       09:37:33 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1\n    logger.go:42:
2026-03-14 09:44:27.189368 | controller |       09:37:33 | watcher-tls/1-deploy-with-tlse | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:44:27.189374 | controller |       09:37:33 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n    logger.go:42: 09:37:33
2026-03-14 09:44:27.189380 | controller |       | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.189387 | controller |       -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.189393 | controller |       | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n    logger.go:42:
2026-03-14 09:44:27.189399 | controller |       09:37:34 | watcher-tls/1-deploy-with-tlse | + '[' 2 == 2 ']'\n    logger.go:42:
2026-03-14 09:44:27.189405 | controller |       09:37:34 | watcher-tls/1-deploy-with-tlse | + grep infra-optim\n    logger.go:42:
2026-03-14 09:44:27.189411 | controller |       09:37:34 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.189417 | controller |       -- openstack endpoint list\n    logger.go:42: 09:37:34 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.189424 | controller |       | ++ grep -c https\n    logger.go:42: 09:37:35 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.189430 | controller |       | + '[' 2 == 2 ']'\n    logger.go:42: 09:37:35 | watcher-tls/1-deploy-with-tlse
2026-03-14 09:44:27.189446 | controller |       | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:44:27.189452 | controller |       \   logger.go:42: 09:37:36 | watcher-tls/1-deploy-with-tlse | + '[' '' == '' ']'\n
2026-03-14 09:44:27.189458 | controller |       \   logger.go:42: 09:37:36 | watcher-tls/1-deploy-with-tlse | + exit 0\n    logger.go:42:
2026-03-14 09:44:27.189464 | controller |       09:37:36 | watcher-tls/1-deploy-with-tlse | test step completed 1-deploy-with-tlse\n
2026-03-14 09:44:27.189471 | controller |       \   logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | starting test step 2-patch-mtls\n
2026-03-14 09:44:27.189477 | controller |       \   logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | running command: [sh -c
2026-03-14 09:44:27.189483 | controller |       set -euxo pipefail\n        oc patch oscp -n $NAMESPACE openstack --type='json'
2026-03-14 09:44:27.189489 | controller |       -p='[{\"op\": \"replace\", \"path\": \"/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode\",
2026-03-14 09:44:27.189496 | controller |       \"value\": \"Request\"}]'\n        ]\n    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.189502 | controller |       | + oc patch oscp -n watcher-kuttl-default openstack --type=json '-p=[{\"op\": \"replace\",
2026-03-14 09:44:27.189508 | controller |       \"path\": \"/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode\", \"value\":
2026-03-14 09:44:27.189557 | controller |       \"Request\"}]'\n    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | openstackcontrolplane.core.openstack.org/openstack
2026-03-14 09:44:27.189568 | controller |       patched\n    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | running command:
2026-03-14 09:44:27.189574 | controller |       [sh -c set -euxo pipefail\n        \n        oc project ${NAMESPACE}\n        #
2026-03-14 09:44:27.189580 | controller |       Get pod names for each watcher service\n        APIPOD=$(oc get pods -l service=watcher-api
2026-03-14 09:44:27.189586 | controller |       -o jsonpath='{.items[0].metadata.name}')\n        APPLIERPOD=$(oc get pods -l service=watcher-applier
2026-03-14 09:44:27.189592 | controller |       -o jsonpath='{.items[0].metadata.name}')\n        DECISIONENGINEPOD=$(oc get pods
2026-03-14 09:44:27.189598 | controller |       -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n        \n
2026-03-14 09:44:27.189603 | controller |       \       # Verify memcached mTLS config parameters in watcher-api config\n        if
2026-03-14 09:44:27.189609 | controller |       [ -n \"${APIPOD}\" ]; then\n          echo \"Checking watcher-api config...\"\n
2026-03-14 09:44:27.189619 | controller |       \         [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.189624 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:44:27.189628 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.189632 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:44:27.189636 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.189640 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:44:27.189645 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.189649 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          # Verify
2026-03-14 09:44:27.189653 | controller |       mTLS config parameters in memcached backend config\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.189684 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:44:27.189688 | controller |       \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.189692 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:44:27.189697 | controller |       \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.189703 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:44:27.189707 | controller |       \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n          exit
2026-03-14 09:44:27.189711 | controller |       1\n        fi\n        \n        # Verify memcached mTLS config parameters in watcher-applier
2026-03-14 09:44:27.189715 | controller |       config\n        if [ -n \"${APPLIERPOD}\" ]; then\n          echo \"Checking watcher-applier
2026-03-14 09:44:27.189720 | controller |       config...\"\n          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.189724 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:44:27.189728 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.189732 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:44:27.189737 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.189741 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:44:27.189753 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.189761 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          [ $(echo
2026-03-14 09:44:27.189765 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.189769 | controller |       | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.189773 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.189777 | controller |       | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.189782 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.189786 | controller |       | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n
2026-03-14 09:44:27.189790 | controller |       \         exit 1\n        fi\n        \n        # Verify memcached mTLS config parameters
2026-03-14 09:44:27.189794 | controller |       in watcher-decision-engine config\n        if [ -n \"${DECISIONENGINEPOD}\" ]; then\n
2026-03-14 09:44:27.189799 | controller |       \         echo \"Checking watcher-decision-engine config...\"\n          [ $(echo
2026-03-14 09:44:27.189803 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.189807 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:44:27.189811 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.189815 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:44:27.189820 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.189824 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:44:27.189828 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.189832 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          [ $(echo
2026-03-14 09:44:27.189837 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.189841 | controller |       | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.189845 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.189849 | controller |       | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.189854 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.189858 | controller |       | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n
2026-03-14 09:44:27.189862 | controller |       \         exit 1\n        fi\n        ]\n    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.189866 | controller |       | + oc project watcher-kuttl-default\n    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.189870 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.189874 | controller |       \   logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api
2026-03-14 09:44:27.189879 | controller |       -o 'jsonpath={.items[0].metadata.name}'\n    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.189883 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.189887 | controller |       | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n
2026-03-14 09:44:27.189891 | controller |       \   logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n
2026-03-14 09:44:27.189895 | controller |       \   logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine
2026-03-14 09:44:27.189902 | controller |       -o 'jsonpath={.items[0].metadata.name}'\n    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.189906 | controller |       | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n    logger.go:42: 09:37:36
2026-03-14 09:44:27.189910 | controller |       | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42:
2026-03-14 09:44:27.189915 | controller |       09:37:36 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n
2026-03-14 09:44:27.189919 | controller |       \   logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n
2026-03-14 09:44:27.189923 | controller |       \   logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile
2026-03-14 09:44:27.189928 | controller |       = /etc/pki/tls/certs/mtls.crt '\n    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.189934 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.189946 | controller |       \   logger.go:42: 09:37:37 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.189953 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.189960 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.189966 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.189972 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.189978 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.189982 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.189986 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.189990 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:44:27.189995 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:44:27.189999 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.190003 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.190011 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.190017 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.190023 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.190029 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.190035 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.190041 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.190047 | controller |       tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:44:27.190053 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]'
2026-03-14 09:44:27.190059 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.190064 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.190070 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.190079 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.190085 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.190091 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.190097 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.190103 | controller |       period = 900\n    logger.go:42: 09:37:37 | watcher-tls/2-patch-mtls | + '[' 0 ==
2026-03-14 09:44:27.190109 | controller |       1 ']'\n    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | running command:
2026-03-14 09:44:27.190116 | controller |       [sh -c set -euxo pipefail\n        \n        oc project ${NAMESPACE}\n        #
2026-03-14 09:44:27.190122 | controller |       Get pod names for each watcher service\n        APIPOD=$(oc get pods -l service=watcher-api
2026-03-14 09:44:27.190132 | controller |       -o jsonpath='{.items[0].metadata.name}')\n        APPLIERPOD=$(oc get pods -l service=watcher-applier
2026-03-14 09:44:27.190138 | controller |       -o jsonpath='{.items[0].metadata.name}')\n        DECISIONENGINEPOD=$(oc get pods
2026-03-14 09:44:27.190144 | controller |       -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n        \n
2026-03-14 09:44:27.190150 | controller |       \       # Verify memcached mTLS config parameters in watcher-api config\n        if
2026-03-14 09:44:27.190155 | controller |       [ -n \"${APIPOD}\" ]; then\n          echo \"Checking watcher-api config...\"\n
2026-03-14 09:44:27.190160 | controller |       \         [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190164 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:44:27.190168 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190172 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:44:27.190184 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190189 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:44:27.190193 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190197 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          # Verify
2026-03-14 09:44:27.190202 | controller |       mTLS config parameters in memcached backend config\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.190206 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:44:27.190210 | controller |       \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.190214 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:44:27.190218 | controller |       \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.190222 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:44:27.190227 | controller |       \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n          exit
2026-03-14 09:44:27.190231 | controller |       1\n        fi\n        \n        # Verify memcached mTLS config parameters in watcher-applier
2026-03-14 09:44:27.190235 | controller |       config\n        if [ -n \"${APPLIERPOD}\" ]; then\n          echo \"Checking watcher-applier
2026-03-14 09:44:27.190239 | controller |       config...\"\n          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190244 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:44:27.190250 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190256 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:44:27.190260 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190264 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:44:27.190268 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190272 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          [ $(echo
2026-03-14 09:44:27.190277 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190281 | controller |       | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.190285 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190303 | controller |       | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.190311 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190315 | controller |       | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n
2026-03-14 09:44:27.190319 | controller |       \         exit 1\n        fi\n        \n        # Verify memcached mTLS config parameters
2026-03-14 09:44:27.190323 | controller |       in watcher-decision-engine config\n        if [ -n \"${DECISIONENGINEPOD}\" ]; then\n
2026-03-14 09:44:27.190327 | controller |       \         echo \"Checking watcher-decision-engine config...\"\n          [ $(echo
2026-03-14 09:44:27.190332 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190336 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:44:27.190340 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190344 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:44:27.190348 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190353 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:44:27.190357 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190361 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          [ $(echo
2026-03-14 09:44:27.190368 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190372 | controller |       | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.190376 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190380 | controller |       | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.190385 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190395 | controller |       | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n
2026-03-14 09:44:27.190400 | controller |       \         exit 1\n        fi\n        ]\n    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.190404 | controller |       | + oc project watcher-kuttl-default\n    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.190408 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.190412 | controller |       \   logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api
2026-03-14 09:44:27.190418 | controller |       -o 'jsonpath={.items[0].metadata.name}'\n    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.190422 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.190427 | controller |       | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n
2026-03-14 09:44:27.190431 | controller |       \   logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n
2026-03-14 09:44:27.190435 | controller |       \   logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine
2026-03-14 09:44:27.190439 | controller |       -o 'jsonpath={.items[0].metadata.name}'\n    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.190444 | controller |       | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n    logger.go:42: 09:37:38
2026-03-14 09:44:27.190450 | controller |       | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42:
2026-03-14 09:44:27.190455 | controller |       09:37:38 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n
2026-03-14 09:44:27.190461 | controller |       \   logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n
2026-03-14 09:44:27.190467 | controller |       \   logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile
2026-03-14 09:44:27.190473 | controller |       = /etc/pki/tls/certs/mtls.crt '\n    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.190479 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.190485 | controller |       \   logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | error: cannot exec into
2026-03-14 09:44:27.190491 | controller |       a container in a completed pod; current phase is Failed\n    logger.go:42: 09:37:38
2026-03-14 09:44:27.190497 | controller |       | watcher-tls/2-patch-mtls | ++ echo\n    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.190503 | controller |       | + '[' 0 == 1 ']'\n    logger.go:42: 09:37:39 | watcher-tls/2-patch-mtls | running
2026-03-14 09:44:27.190509 | controller |       command: [sh -c set -euxo pipefail\n        \n        oc project ${NAMESPACE}\n
2026-03-14 09:44:27.190515 | controller |       \       # Get pod names for each watcher service\n        APIPOD=$(oc get pods -l
2026-03-14 09:44:27.190521 | controller |       service=watcher-api -o jsonpath='{.items[0].metadata.name}')\n        APPLIERPOD=$(oc
2026-03-14 09:44:27.190527 | controller |       get pods -l service=watcher-applier -o jsonpath='{.items[0].metadata.name}')\n        DECISIONENGINEPOD=$(oc
2026-03-14 09:44:27.190537 | controller |       get pods -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n
2026-03-14 09:44:27.190544 | controller |       \       \n        # Verify memcached mTLS config parameters in watcher-api config\n
2026-03-14 09:44:27.190550 | controller |       \       if [ -n \"${APIPOD}\" ]; then\n          echo \"Checking watcher-api config...\"\n
2026-03-14 09:44:27.190555 | controller |       \         [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190561 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:44:27.190568 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190574 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:44:27.190580 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190586 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:44:27.190592 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190599 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          # Verify
2026-03-14 09:44:27.190605 | controller |       mTLS config parameters in memcached backend config\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.190614 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:44:27.190620 | controller |       \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.190626 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:44:27.190632 | controller |       \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.190638 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:44:27.190654 | controller |       \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n          exit
2026-03-14 09:44:27.190661 | controller |       1\n        fi\n        \n        # Verify memcached mTLS config parameters in watcher-applier
2026-03-14 09:44:27.190667 | controller |       config\n        if [ -n \"${APPLIERPOD}\" ]; then\n          echo \"Checking watcher-applier
2026-03-14 09:44:27.190673 | controller |       config...\"\n          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190679 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:44:27.190685 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190691 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:44:27.190697 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190704 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:44:27.190714 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190720 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          [ $(echo
2026-03-14 09:44:27.190726 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190732 | controller |       | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.190738 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190743 | controller |       | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.190749 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190755 | controller |       | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n
2026-03-14 09:44:27.190761 | controller |       \         exit 1\n        fi\n        \n        # Verify memcached mTLS config parameters
2026-03-14 09:44:27.190767 | controller |       in watcher-decision-engine config\n        if [ -n \"${DECISIONENGINEPOD}\" ]; then\n
2026-03-14 09:44:27.190773 | controller |       \         echo \"Checking watcher-decision-engine config...\"\n          [ $(echo
2026-03-14 09:44:27.190779 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190785 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:44:27.190792 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190798 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:44:27.190804 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190811 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:44:27.190817 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190828 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          [ $(echo
2026-03-14 09:44:27.190834 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190840 | controller |       | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.190847 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190853 | controller |       | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.190859 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.190865 | controller |       | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n
2026-03-14 09:44:27.190871 | controller |       \         exit 1\n        fi\n        ]\n    logger.go:42: 09:37:39 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.190878 | controller |       | + oc project watcher-kuttl-default\n    logger.go:42: 09:37:39 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.190884 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.190891 | controller |       \   logger.go:42: 09:37:39 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api
2026-03-14 09:44:27.190897 | controller |       -o 'jsonpath={.items[0].metadata.name}'\n    logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.190903 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.190910 | controller |       | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n
2026-03-14 09:44:27.190917 | controller |       \   logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n
2026-03-14 09:44:27.190935 | controller |       \   logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine
2026-03-14 09:44:27.190943 | controller |       -o 'jsonpath={.items[0].metadata.name}'\n    logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.190949 | controller |       | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n    logger.go:42: 09:37:40
2026-03-14 09:44:27.190955 | controller |       | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42:
2026-03-14 09:44:27.190960 | controller |       09:37:40 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n
2026-03-14 09:44:27.190965 | controller |       \   logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n
2026-03-14 09:44:27.190969 | controller |       \   logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile
2026-03-14 09:44:27.190973 | controller |       = /etc/pki/tls/certs/mtls.crt '\n    logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.191011 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.191016 | controller |       \   logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls | error: unable to upgrade
2026-03-14 09:44:27.191020 | controller |       connection: container not found (\"watcher-api\")\n    logger.go:42: 09:37:40 |
2026-03-14 09:44:27.191024 | controller |       watcher-tls/2-patch-mtls | ++ echo\n    logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.191028 | controller |       | + '[' 0 == 1 ']'\n    logger.go:42: 09:37:41 | watcher-tls/2-patch-mtls | running
2026-03-14 09:44:27.191033 | controller |       command: [sh -c set -euxo pipefail\n        \n        oc project ${NAMESPACE}\n
2026-03-14 09:44:27.191037 | controller |       \       # Get pod names for each watcher service\n        APIPOD=$(oc get pods -l
2026-03-14 09:44:27.191041 | controller |       service=watcher-api -o jsonpath='{.items[0].metadata.name}')\n        APPLIERPOD=$(oc
2026-03-14 09:44:27.191045 | controller |       get pods -l service=watcher-applier -o jsonpath='{.items[0].metadata.name}')\n        DECISIONENGINEPOD=$(oc
2026-03-14 09:44:27.191049 | controller |       get pods -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n
2026-03-14 09:44:27.191058 | controller |       \       \n        # Verify memcached mTLS config parameters in watcher-api config\n
2026-03-14 09:44:27.191062 | controller |       \       if [ -n \"${APIPOD}\" ]; then\n          echo \"Checking watcher-api config...\"\n
2026-03-14 09:44:27.191066 | controller |       \         [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.191070 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:44:27.191074 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.191079 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:44:27.191083 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.191087 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:44:27.191091 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.191095 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          # Verify
2026-03-14 09:44:27.191099 | controller |       mTLS config parameters in memcached backend config\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.191104 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:44:27.191108 | controller |       \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.191112 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:44:27.191116 | controller |       \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.191123 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:44:27.191127 | controller |       \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n          exit
2026-03-14 09:44:27.191131 | controller |       1\n        fi\n        \n        # Verify memcached mTLS config parameters in watcher-applier
2026-03-14 09:44:27.191136 | controller |       config\n        if [ -n \"${APPLIERPOD}\" ]; then\n          echo \"Checking watcher-applier
2026-03-14 09:44:27.191140 | controller |       config...\"\n          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.191144 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:44:27.191177 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.191182 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:44:27.191186 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.191190 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:44:27.191201 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.191206 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          [ $(echo
2026-03-14 09:44:27.191210 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.191214 | controller |       | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.191218 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.191222 | controller |       | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.191251 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.191260 | controller |       | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n
2026-03-14 09:44:27.191270 | controller |       \         exit 1\n        fi\n        \n        # Verify memcached mTLS config parameters
2026-03-14 09:44:27.191276 | controller |       in watcher-decision-engine config\n        if [ -n \"${DECISIONENGINEPOD}\" ]; then\n
2026-03-14 09:44:27.191282 | controller |       \         echo \"Checking watcher-decision-engine config...\"\n          [ $(echo
2026-03-14 09:44:27.191290 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.191318 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:44:27.191325 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.191331 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:44:27.191337 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.191343 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:44:27.191349 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.191355 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          [ $(echo
2026-03-14 09:44:27.191361 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.191365 | controller |       | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.191369 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.191374 | controller |       | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.191378 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.191382 | controller |       | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n
2026-03-14 09:44:27.191389 | controller |       \         exit 1\n        fi\n        ]\n    logger.go:42: 09:37:41 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.191393 | controller |       | + oc project watcher-kuttl-default\n    logger.go:42: 09:37:41 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.191398 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.191402 | controller |       \   logger.go:42: 09:37:41 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api
2026-03-14 09:44:27.191406 | controller |       -o 'jsonpath={.items[0].metadata.name}'\n    logger.go:42: 09:37:41 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.191410 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:37:41 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.191415 | controller |       | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n
2026-03-14 09:44:27.191420 | controller |       \   logger.go:42: 09:37:41 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n
2026-03-14 09:44:27.191426 | controller |       \   logger.go:42: 09:37:41 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine
2026-03-14 09:44:27.191432 | controller |       -o 'jsonpath={.items[0].metadata.name}'\n    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.191438 | controller |       | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n    logger.go:42: 09:37:42
2026-03-14 09:44:27.191444 | controller |       | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42:
2026-03-14 09:44:27.191450 | controller |       09:37:42 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n
2026-03-14 09:44:27.191456 | controller |       \   logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n
2026-03-14 09:44:27.191462 | controller |       \   logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.191468 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:44:27.191476 | controller |       09:37:42 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt
2026-03-14 09:44:27.191491 | controller |       '\n    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.191496 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.191500 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.191504 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.191509 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.191513 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.191517 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.191521 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.191526 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.191530 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.191538 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.191542 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.191546 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.191551 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.191555 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.191559 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.191563 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.191567 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.191572 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.191576 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.191580 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.191584 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.191589 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.191593 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.191597 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.191601 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.191606 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.191610 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.191615 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.191621 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.191629 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.191636 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.191641 | controller |       period = 900\n    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.191647 | controller |       1 ']'\n    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile
2026-03-14 09:44:27.191653 | controller |       = /etc/pki/tls/private/mtls.key'\n    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.191659 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.191665 | controller |       \   logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.191671 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.191676 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.191682 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.191695 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.191700 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.191704 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.191709 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.191714 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.191718 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.191722 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.191726 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.191730 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.191735 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.191739 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.191743 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.191747 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.191752 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.191756 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.191760 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.191764 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.191771 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.191775 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.191779 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.191784 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.191790 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.191794 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.191798 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.191803 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.191807 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.191811 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.191815 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.191821 | controller |       period = 900\n    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.191825 | controller |       1 ']'\n    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile
2026-03-14 09:44:27.191830 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'\n    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.191834 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.191838 | controller |       \   logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.191842 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.191847 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.191851 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.191855 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.191859 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.191863 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.191868 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.191877 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.191882 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.191886 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.191891 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.191895 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.191899 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.191903 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.191910 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.191915 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.191919 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.191923 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.191928 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.191932 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.191939 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.191945 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.191951 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.191956 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.191963 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.191969 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.191975 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.191981 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.191987 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.191993 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.191999 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.192005 | controller |       period = 900\n    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.192011 | controller |       1 ']'\n    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled
2026-03-14 09:44:27.192018 | controller |       = true'\n    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.192024 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:44:27.192031 | controller |       09:37:43 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:44:27.192037 | controller |       transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.192043 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.192050 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.192056 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.192062 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.192069 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.192075 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.192082 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.192088 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.192095 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.192101 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.192107 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.192123 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.192130 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.192136 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.192146 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.192153 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.192159 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.192165 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.192171 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.192177 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.192186 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.192192 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.192198 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.192205 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.192213 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.192219 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.192224 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.192230 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.192236 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.192242 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.192248 | controller |       period = 900\n    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.192254 | controller |       1 ']'\n    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n
2026-03-14 09:44:27.192261 | controller |       \   logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.192267 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:44:27.192327 | controller |       09:37:43 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:44:27.192370 | controller |       transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.192378 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.192384 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.192391 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.192397 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.192407 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.192413 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.192419 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.192426 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.192432 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.192483 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.192489 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.192494 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.192498 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.192502 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.192507 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.192511 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.192524 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.192530 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.192534 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.192538 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.192543 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.192547 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.192551 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.192556 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.192560 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.192564 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.192569 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.192573 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.192577 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.192582 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.192586 | controller |       period = 900\n    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.192590 | controller |       1 ']'\n    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n
2026-03-14 09:44:27.192595 | controller |       \   logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.192604 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:44:27.192610 | controller |       09:37:43 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:44:27.192616 | controller |       transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.192623 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.192629 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.192635 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.192641 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.192648 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.192656 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.192662 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.192668 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.192674 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.192683 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.192690 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.192697 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.192703 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.192710 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.192717 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.192723 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.192729 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.192735 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.192742 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.192748 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.192754 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.192771 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.192779 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.192785 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.192791 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.192798 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.192803 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.192810 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.192816 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.192822 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.192829 | controller |       period = 900\n    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.192835 | controller |       1 ']'\n    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n
2026-03-14 09:44:27.192842 | controller |       \   logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.192847 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:44:27.192853 | controller |       09:37:43 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:44:27.192859 | controller |       transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.192865 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.192979 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.192991 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.192998 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.193004 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.193028 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.193039 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.193047 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.193053 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.193059 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.193065 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.193071 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.193077 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.193083 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.193089 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.193095 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.193102 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.193108 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.193114 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.193121 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.193128 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.193134 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.193140 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.193147 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.193153 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.193159 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.193175 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.193182 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.193188 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.193194 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.193202 | controller |       period = 900\n    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.193208 | controller |       1 ']'\n    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-applier-0
2026-03-14 09:44:27.193214 | controller |       ']'\n    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-applier
2026-03-14 09:44:27.193222 | controller |       config...'\n    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | Checking watcher-applier
2026-03-14 09:44:27.193227 | controller |       config...\n    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile
2026-03-14 09:44:27.193233 | controller |       = /etc/pki/tls/certs/mtls.crt '\n    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.193238 | controller |       | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.193244 | controller |       \   logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.193250 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.193255 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.193261 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.193269 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.193275 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.193280 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.193286 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.193310 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.193316 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.193322 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.193327 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.193333 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.193338 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.193344 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.193349 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.193355 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.193360 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.193366 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.193371 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.193377 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.193382 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:44:27.193388 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.193394 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.193399 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.193405 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.193410 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.193418 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.193431 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.193437 | controller |       period = 900\n    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.193442 | controller |       1 ']'\n    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile
2026-03-14 09:44:27.193448 | controller |       = /etc/pki/tls/private/mtls.key'\n    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.193454 | controller |       | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.193459 | controller |       \   logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.193465 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.193470 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.193476 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.193482 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.193487 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.193493 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.193499 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.193504 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.193510 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.193516 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.193522 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.193527 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.193533 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.193539 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.193544 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.193550 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.193555 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.193561 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.193567 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.193572 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.193578 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:44:27.193586 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.193591 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.193601 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.193606 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.193612 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.193618 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.193624 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.193630 | controller |       period = 900\n    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.193636 | controller |       1 ']'\n    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile
2026-03-14 09:44:27.193642 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'\n    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.193648 | controller |       | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.193654 | controller |       \   logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.193685 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.193692 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.193698 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.193704 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.193710 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.193715 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.193721 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.193727 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.193732 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.193738 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.193743 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.193749 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.193755 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.193760 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.193766 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.193771 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.193777 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.193782 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.193788 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.193794 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.193799 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:44:27.193807 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.193813 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.193818 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.193824 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.193830 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.193835 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.193843 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.193849 | controller |       period = 900\n    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.193855 | controller |       1 ']'\n    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.193860 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:44:27.193866 | controller |       09:37:44 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled = true'\n
2026-03-14 09:44:27.193872 | controller |       \   logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.193877 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.193883 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.193889 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.193894 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.193900 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.193906 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.193921 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.193928 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.193935 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.193942 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.193947 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.193953 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.193959 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.193964 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.193970 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.193975 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.193983 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.193989 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.193995 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.194002 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.194008 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:44:27.194014 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.194019 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.194025 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.194030 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.194036 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.194042 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.194047 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.194053 | controller |       period = 900\n    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.194059 | controller |       1 ']'\n    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n
2026-03-14 09:44:27.194064 | controller |       \   logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.194070 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:44:27.194075 | controller |       09:37:44 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:44:27.194081 | controller |       transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.194087 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.194092 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.194098 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.194103 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.194109 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.194115 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.194120 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.194126 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.194131 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.194137 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.194142 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.194148 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.194160 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.194166 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.194172 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.194177 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.194185 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.194191 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.194196 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.194202 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:44:27.194210 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.194215 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.194222 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.194228 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.194234 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.194239 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.194245 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.194250 | controller |       period = 900\n    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.194256 | controller |       1 ']'\n    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n
2026-03-14 09:44:27.194262 | controller |       \   logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.194267 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:44:27.194273 | controller |       09:37:45 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:44:27.194279 | controller |       transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.194285 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.194308 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.194315 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.194321 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.194327 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.194333 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.194340 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.194346 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.194352 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.194358 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.194365 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.194371 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.194381 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.194390 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.194396 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.194402 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.194408 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.194414 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.194420 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.194435 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:44:27.194442 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.194449 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.194455 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.194461 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.194467 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.194473 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.194479 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.194485 | controller |       period = 900\n    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.194491 | controller |       1 ']'\n    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n
2026-03-14 09:44:27.194497 | controller |       \   logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.194504 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:44:27.194509 | controller |       09:37:45 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:44:27.194515 | controller |       transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.194521 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.194527 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.194533 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.194539 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.194546 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.194552 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.194558 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.194564 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.194570 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.194576 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.194582 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.194591 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.194597 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.194604 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.194610 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.194616 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.194622 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.194628 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.194637 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.194643 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:44:27.194649 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.194655 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.194661 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.194667 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.194673 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.194679 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.194694 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.194700 | controller |       period = 900\n    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.194705 | controller |       1 ']'\n    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.194712 | controller |       ']'\n    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-decision-engine
2026-03-14 09:44:27.194718 | controller |       config...'\n    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | Checking watcher-decision-engine
2026-03-14 09:44:27.194724 | controller |       config...\n    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile
2026-03-14 09:44:27.194730 | controller |       = /etc/pki/tls/certs/mtls.crt '\n    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.194736 | controller |       | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.194742 | controller |       \   logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | error: Internal error occurred:
2026-03-14 09:44:27.194748 | controller |       error executing command in container: container is not created or running\n    logger.go:42:
2026-03-14 09:44:27.194754 | controller |       09:37:45 | watcher-tls/2-patch-mtls | ++ echo\n    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.194760 | controller |       | + '[' 0 == 1 ']'\n    logger.go:42: 09:37:46 | watcher-tls/2-patch-mtls | running
2026-03-14 09:44:27.194766 | controller |       command: [sh -c set -euxo pipefail\n        \n        oc project ${NAMESPACE}\n
2026-03-14 09:44:27.194772 | controller |       \       # Get pod names for each watcher service\n        APIPOD=$(oc get pods -l
2026-03-14 09:44:27.194781 | controller |       service=watcher-api -o jsonpath='{.items[0].metadata.name}')\n        APPLIERPOD=$(oc
2026-03-14 09:44:27.194787 | controller |       get pods -l service=watcher-applier -o jsonpath='{.items[0].metadata.name}')\n        DECISIONENGINEPOD=$(oc
2026-03-14 09:44:27.194793 | controller |       get pods -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n
2026-03-14 09:44:27.194800 | controller |       \       \n        # Verify memcached mTLS config parameters in watcher-api config\n
2026-03-14 09:44:27.194806 | controller |       \       if [ -n \"${APIPOD}\" ]; then\n          echo \"Checking watcher-api config...\"\n
2026-03-14 09:44:27.194812 | controller |       \         [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.194818 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:44:27.194824 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.194830 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:44:27.194836 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.194842 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:44:27.194848 | controller |       $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.194854 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          # Verify
2026-03-14 09:44:27.194860 | controller |       mTLS config parameters in memcached backend config\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.194866 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:44:27.194872 | controller |       \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.194878 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:44:27.194884 | controller |       \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.194890 | controller |       -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c
2026-03-14 09:44:27.194896 | controller |       \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n          exit
2026-03-14 09:44:27.194902 | controller |       1\n        fi\n        \n        # Verify memcached mTLS config parameters in watcher-applier
2026-03-14 09:44:27.194911 | controller |       config\n        if [ -n \"${APPLIERPOD}\" ]; then\n          echo \"Checking watcher-applier
2026-03-14 09:44:27.194917 | controller |       config...\"\n          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.194923 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:44:27.194929 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.194935 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:44:27.194942 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.194948 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:44:27.194962 | controller |       $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.194969 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          [ $(echo
2026-03-14 09:44:27.194976 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.194981 | controller |       | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.194987 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.194993 | controller |       | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.195000 | controller |       $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.195006 | controller |       | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n
2026-03-14 09:44:27.195018 | controller |       \         exit 1\n        fi\n        \n        # Verify memcached mTLS config parameters
2026-03-14 09:44:27.195025 | controller |       in watcher-decision-engine config\n        if [ -n \"${DECISIONENGINEPOD}\" ]; then\n
2026-03-14 09:44:27.195031 | controller |       \         echo \"Checking watcher-decision-engine config...\"\n          [ $(echo
2026-03-14 09:44:27.195036 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.195042 | controller |       | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n          [
2026-03-14 09:44:27.195048 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.195057 | controller |       | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n          [
2026-03-14 09:44:27.195063 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.195069 | controller |       | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n          [
2026-03-14 09:44:27.195074 | controller |       $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.195080 | controller |       | grep -c \"memcache_tls_enabled = true\") == 1 ]\n        \n          [ $(echo
2026-03-14 09:44:27.195086 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.195092 | controller |       | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.195099 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.195105 | controller |       | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n          [ $(echo
2026-03-14 09:44:27.195111 | controller |       $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.195117 | controller |       | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n        else\n
2026-03-14 09:44:27.195124 | controller |       \         exit 1\n        fi\n        ]\n    logger.go:42: 09:37:46 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.195130 | controller |       | + oc project watcher-kuttl-default\n    logger.go:42: 09:37:46 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.195136 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.195142 | controller |       \   logger.go:42: 09:37:46 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api
2026-03-14 09:44:27.195148 | controller |       -o 'jsonpath={.items[0].metadata.name}'\n    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.195153 | controller |       | + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.195159 | controller |       | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n
2026-03-14 09:44:27.195165 | controller |       \   logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n
2026-03-14 09:44:27.195171 | controller |       \   logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine
2026-03-14 09:44:27.195177 | controller |       -o 'jsonpath={.items[0].metadata.name}'\n    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.195184 | controller |       | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n    logger.go:42: 09:37:47
2026-03-14 09:44:27.195190 | controller |       | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42:
2026-03-14 09:44:27.195196 | controller |       09:37:47 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n
2026-03-14 09:44:27.195202 | controller |       \   logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n
2026-03-14 09:44:27.195208 | controller |       \   logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile
2026-03-14 09:44:27.195216 | controller |       = /etc/pki/tls/certs/mtls.crt '\n    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.195223 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.195237 | controller |       \   logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.195244 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.195250 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.195256 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.195262 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.195268 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.195275 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.195281 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.195290 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.195313 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.195320 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.195326 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.195332 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.195338 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.195344 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.195350 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.195356 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.195362 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.195368 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.195374 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.195380 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.195386 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.195392 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.195398 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.195404 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.195410 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.195416 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.195422 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.195428 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.195434 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.195446 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.195452 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.195459 | controller |       period = 900\n    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.195465 | controller |       1 ']'\n    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile
2026-03-14 09:44:27.195471 | controller |       = /etc/pki/tls/private/mtls.key'\n    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.195478 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.195485 | controller |       \   logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.195491 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.195497 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.195503 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.195518 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.195525 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.195532 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.195538 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.195544 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.195551 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.195557 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.195563 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.195569 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.195576 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.195582 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.195588 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.195595 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.195601 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.195607 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.195614 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.195620 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.195626 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.195632 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.195638 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.195644 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.195654 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.195660 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.195666 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.195672 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.195678 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.195685 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.195691 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.195697 | controller |       period = 900\n    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.195703 | controller |       1 ']'\n    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile
2026-03-14 09:44:27.195710 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'\n    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.195716 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.195722 | controller |       \   logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.195728 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.195734 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.195740 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.195747 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.195753 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.195759 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.195765 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.195779 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.195787 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.195793 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.195799 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.195805 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.195811 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.195818 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.195824 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.195830 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.195836 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.195842 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.195848 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.195857 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.195863 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.195870 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.195876 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.195887 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.195894 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.195900 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.195906 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.195912 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.195919 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.195925 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.195931 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.195938 | controller |       period = 900\n    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.195944 | controller |       1 ']'\n    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled
2026-03-14 09:44:27.195952 | controller |       = true'\n    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.195959 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:44:27.195965 | controller |       09:37:48 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:44:27.195971 | controller |       transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.195977 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.195983 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.195990 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.195996 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.196002 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.196009 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.196015 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.196021 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.196027 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.196033 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.196040 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.196054 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.196061 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.196067 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.196076 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.196082 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.196088 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.196094 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.196100 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.196106 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.196112 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.196119 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.196125 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.196131 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.196138 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.196144 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.196150 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.196156 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.196162 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.196169 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.196175 | controller |       period = 900\n    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.196181 | controller |       1 ']'\n    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n
2026-03-14 09:44:27.196187 | controller |       \   logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.196193 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:44:27.196199 | controller |       09:37:48 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:44:27.196206 | controller |       transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.196212 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.196218 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.196224 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.196231 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.196237 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.196243 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.196249 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.196256 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.196262 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.196270 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.196277 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.196283 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.196304 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.196313 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.196322 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.196328 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.196341 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.196347 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.196353 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.196359 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.196365 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.196371 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.196376 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.196382 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.196388 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.196393 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.196399 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.196405 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.196411 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.196417 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.196422 | controller |       period = 900\n    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.196428 | controller |       1 ']'\n    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n
2026-03-14 09:44:27.196434 | controller |       \   logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.196440 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:44:27.196446 | controller |       09:37:48 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:44:27.196452 | controller |       transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.196458 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.196464 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.196471 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.196477 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.196486 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.196494 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.196500 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.196506 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.196512 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.196518 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.196524 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.196530 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.196536 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.196545 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.196551 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.196557 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.196563 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.196569 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.196575 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.196581 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.196587 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.196601 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.196607 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.196614 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.196620 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.196626 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.196632 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.196638 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.196643 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.196649 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.196655 | controller |       period = 900\n    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.196661 | controller |       1 ']'\n    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n
2026-03-14 09:44:27.196667 | controller |       \   logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.196673 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:44:27.196682 | controller |       09:37:48 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:44:27.196688 | controller |       transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.196695 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.196703 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.196710 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.196716 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.196722 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.196728 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.196733 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.196739 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.196745 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.196751 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.196758 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.196764 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.196770 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.196776 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.196782 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.196788 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.196795 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.196801 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.196807 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.196813 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.196819 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.196825 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.196831 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.196837 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.196843 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.196849 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.196864 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.196871 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.196877 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.196883 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.196889 | controller |       period = 900\n    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.196895 | controller |       1 ']'\n    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-applier-0
2026-03-14 09:44:27.196902 | controller |       ']'\n    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-applier
2026-03-14 09:44:27.196911 | controller |       config...'\n    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | Checking watcher-applier
2026-03-14 09:44:27.196917 | controller |       config...\n    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile
2026-03-14 09:44:27.196923 | controller |       = /etc/pki/tls/certs/mtls.crt '\n    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.196929 | controller |       | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.196935 | controller |       \   logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.196941 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.196948 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.196954 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.196960 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.196966 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.196972 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.196979 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.196988 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.196994 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.197000 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.197006 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.197012 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.197018 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.197025 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.197031 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.197037 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.197043 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.197050 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.197056 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.197063 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.197069 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:44:27.197075 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.197081 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.197090 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.197097 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.197104 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.197112 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.197126 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.197133 | controller |       period = 900\n    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.197139 | controller |       1 ']'\n    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile
2026-03-14 09:44:27.197144 | controller |       = /etc/pki/tls/private/mtls.key'\n    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.197150 | controller |       | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.197156 | controller |       \   logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.197161 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.197167 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.197173 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.197179 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.197186 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.197192 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.197198 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.197204 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.197210 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.197216 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.197222 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.197228 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.197234 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.197240 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.197246 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.197252 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.197259 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.197265 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.197271 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.197277 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.197283 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:44:27.197305 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.197314 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.197325 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.197351 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.197358 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.197364 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.197370 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.197376 | controller |       period = 900\n    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.197382 | controller |       1 ']'\n    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile
2026-03-14 09:44:27.197388 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'\n    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.197394 | controller |       | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.197399 | controller |       \   logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.197414 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.197422 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.197428 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.197434 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.197440 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.197446 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.197452 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.197458 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.197465 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.197471 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.197477 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.197483 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.197489 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.197495 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.197501 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.197507 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.197513 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.197519 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.197528 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.197535 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.197541 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:44:27.197549 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.197555 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.197562 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.197568 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.197574 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.197580 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.197586 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.197592 | controller |       period = 900\n    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.197598 | controller |       1 ']'\n    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled
2026-03-14 09:44:27.197604 | controller |       = true'\n    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.197610 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:44:27.197616 | controller |       09:37:49 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:44:27.197622 | controller |       transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.197631 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.197637 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.197643 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.197649 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.197655 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.197669 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.197698 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.197705 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.197711 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.197717 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.197723 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.197729 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.197736 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.197742 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.197747 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.197753 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.197759 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.197765 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.197775 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.197781 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:44:27.197787 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.197796 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.197803 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.197809 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.197814 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.197820 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.197827 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.197833 | controller |       period = 900\n    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.197839 | controller |       1 ']'\n    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n
2026-03-14 09:44:27.197845 | controller |       \   logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.197851 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:44:27.197857 | controller |       09:37:50 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:44:27.197863 | controller |       transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.197872 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.197878 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.197884 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.197890 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.197896 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.197902 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.197908 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.197914 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.197920 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.197926 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.197932 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.197938 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.197953 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.197960 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.197966 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.197987 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.197997 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.198003 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.198009 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.198015 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:44:27.198021 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.198027 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.198033 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.198039 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.198045 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.198051 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.198057 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.198063 | controller |       period = 900\n    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.198069 | controller |       1 ']'\n    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n
2026-03-14 09:44:27.198076 | controller |       \   logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.198082 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:44:27.198088 | controller |       09:37:50 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher
2026-03-14 09:44:27.198094 | controller |       transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.198100 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.198106 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.198112 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.198118 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.198124 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.198130 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.198136 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.198142 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.198148 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.198154 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.198160 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.198166 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.198172 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.198180 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.198186 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.198192 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.198198 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.198204 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.198210 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.198223 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:44:27.198230 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.198236 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.198242 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.198248 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.198255 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.198261 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.198268 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.198274 | controller |       period = 900\n    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.198280 | controller |       1 ']'\n    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.198287 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:44:27.198312 | controller |       09:37:50 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n
2026-03-14 09:44:27.198319 | controller |       \   logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.198325 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.198332 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.198341 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.198348 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.198354 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.198360 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.198366 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.198373 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.198379 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.198385 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.198391 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.198397 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.198406 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.198415 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.198421 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.198427 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.198434 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.198440 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.198446 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.198454 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.198461 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]'
2026-03-14 09:44:27.198467 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.198474 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.198480 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.198486 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.198492 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.198499 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.198513 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.198520 | controller |       period = 900\n    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.198527 | controller |       1 ']'\n    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.198533 | controller |       ']'\n    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-decision-engine
2026-03-14 09:44:27.198539 | controller |       config...'\n    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | Checking watcher-decision-engine
2026-03-14 09:44:27.198545 | controller |       config...\n    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile
2026-03-14 09:44:27.198552 | controller |       = /etc/pki/tls/certs/mtls.crt '\n    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.198557 | controller |       | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.198563 | controller |       \   logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.198570 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.198576 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.198582 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.198589 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.198595 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.198602 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.198611 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.198617 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.198623 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.198629 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.198635 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.198641 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.198647 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.198653 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.198659 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.198666 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.198672 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.198678 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.198685 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.198691 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.198698 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.198704 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.198710 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.198717 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.198723 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.198730 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.198736 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.198742 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.198749 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.198755 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.198770 | controller |       period = 900\n    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.198778 | controller |       1 ']'\n    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile
2026-03-14 09:44:27.198785 | controller |       = /etc/pki/tls/private/mtls.key'\n    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.198791 | controller |       | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.198797 | controller |       \   logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.198804 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.198810 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.198817 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.198828 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.198835 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.198842 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.198848 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.198857 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.198863 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.198869 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.198875 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.198882 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.198888 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.198894 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.198900 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.198907 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.198913 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.198919 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.198926 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.198932 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.198938 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.198945 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.198951 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.198957 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.198964 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.198970 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.198977 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.198983 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.198989 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.198996 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.199002 | controller |       period = 900\n    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.199011 | controller |       1 ']'\n    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile
2026-03-14 09:44:27.199017 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'\n    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls
2026-03-14 09:44:27.199024 | controller |       | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.199033 | controller |       \   logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.199047 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.199054 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.199060 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.199066 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.199073 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.199079 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.199085 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.199091 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.199097 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.199104 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.199110 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.199116 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.199122 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.199129 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.199135 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.199141 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.199148 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.199154 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.199160 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.199167 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.199173 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.199180 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.199186 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.199193 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.199199 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.199206 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.199215 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.199222 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.199228 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.199234 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.199243 | controller |       period = 900\n    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.199250 | controller |       1 ']'\n    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled
2026-03-14 09:44:27.199256 | controller |       = true'\n    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:44:27.199262 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.199269 | controller |       \   logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.199275 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.199282 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.199306 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.199325 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.199332 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.199339 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.199346 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.199352 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.199359 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.199365 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.199372 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.199378 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.199385 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.199391 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.199398 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.199404 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.199410 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.199417 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.199423 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.199433 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.199440 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.199446 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.199453 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.199459 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.199466 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.199472 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.199479 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.199488 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.199494 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.199501 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.199508 | controller |       period = 900\n    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.199514 | controller |       1 ']'\n    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n
2026-03-14 09:44:27.199521 | controller |       \   logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:44:27.199529 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.199536 | controller |       \   logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.199542 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.199549 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.199555 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.199562 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.199571 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.199578 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.199584 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.199591 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.199605 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.199612 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.199619 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.199625 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.199631 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.199640 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.199647 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.199653 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.199659 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.199665 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.199671 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.199677 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.199685 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.199691 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.199699 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.199705 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.199711 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.199717 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.199723 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.199729 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.199734 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.199740 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.199746 | controller |       period = 900\n    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.199751 | controller |       1 ']'\n    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n
2026-03-14 09:44:27.199757 | controller |       \   logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:44:27.199762 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.199768 | controller |       \   logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.199774 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.199779 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.199785 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.199790 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.199796 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.199802 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.199808 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.199814 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.199819 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.199825 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.199831 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.199837 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.199843 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.199849 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.199862 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.199870 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.199876 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.199882 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.199891 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.199897 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.199903 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.199910 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.199916 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.199922 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.199928 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.199935 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.199941 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.199947 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.199954 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.199960 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.199966 | controller |       period = 900\n    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.199973 | controller |       1 ']'\n    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n
2026-03-14 09:44:27.199980 | controller |       \   logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:44:27.199986 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.199992 | controller |       \   logger.go:42: 09:37:52 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path
2026-03-14 09:44:27.199998 | controller |       = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.200004 | controller |       control_exchange = watcher debug = True '#' empty notification_level means that
2026-03-14 09:44:27.200010 | controller |       no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.200016 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.200022 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.200028 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.200034 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.200040 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.200046 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.200052 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.200061 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.200067 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.200074 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.200080 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.200090 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.200096 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.200102 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.200108 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.200114 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.200129 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.200136 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.200143 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.200149 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type
2026-03-14 09:44:27.200156 | controller |       = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.200162 | controller |       = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]'
2026-03-14 09:44:27.200168 | controller |       endpoint_type = internal region_name = regionOne '[keystone_client]' interface =
2026-03-14 09:44:27.200174 | controller |       internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.200181 | controller |       = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]'
2026-03-14 09:44:27.200187 | controller |       interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.200194 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.200200 | controller |       period = 900\n    logger.go:42: 09:37:52 | watcher-tls/2-patch-mtls | + '[' 1 ==
2026-03-14 09:44:27.200206 | controller |       1 ']'\n    logger.go:42: 09:37:52 | watcher-tls/2-patch-mtls | test step completed
2026-03-14 09:44:27.200213 | controller |       2-patch-mtls\n    logger.go:42: 09:37:52 | watcher-tls/3-disable-podlevel-tls |
2026-03-14 09:44:27.200219 | controller |       starting test step 3-disable-podlevel-tls\n    logger.go:42: 09:37:52 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200226 | controller |       | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json'
2026-03-14 09:44:27.200232 | controller |       -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/tls/api\", \"value\":{
2026-03-14 09:44:27.200239 | controller |       \"internal\": {}, \"public\": {} }}]'\n        ]\n    logger.go:42: 09:37:52 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200245 | controller |       | watcher.watcher.openstack.org/watcher-kuttl patched\n    logger.go:42: 09:37:52
2026-03-14 09:44:27.200252 | controller |       | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:44:27.200258 | controller |       \       oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.200267 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n ${NAMESPACE}
2026-03-14 09:44:27.200274 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep
2026-03-14 09:44:27.200280 | controller |       watcher| awk '{print $1}')\n        [ $(oc get -n ${NAMESPACE} keystoneservice watcher
2026-03-14 09:44:27.200302 | controller |       -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE}
2026-03-14 09:44:27.200313 | controller |       watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n        # check that
2026-03-14 09:44:27.200320 | controller |       watcher internal endpoint does not use https\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:44:27.200326 | controller |       -- openstack endpoint list | grep infra-optim | grep internal  | [ $(grep -c https)
2026-03-14 09:44:27.200336 | controller |       == 0 ]\n        # check that watcher public endpoint does use https\n        oc
2026-03-14 09:44:27.200342 | controller |       exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim
2026-03-14 09:44:27.200348 | controller |       | grep public  | [ $(grep -c https) == 1 ]\n        # If we are running the container
2026-03-14 09:44:27.200355 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:44:27.200361 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:44:27.200368 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:44:27.200374 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:44:27.200381 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:44:27.200388 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:44:27.200394 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:44:27.200403 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:44:27.200409 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:44:27.200416 | controller |       \       ]\n    logger.go:42: 09:37:52 | watcher-tls/3-disable-podlevel-tls | + oc
2026-03-14 09:44:27.200422 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.200427 | controller |       -c Name -c Type\n    logger.go:42: 09:37:52 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200442 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:37:54 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200450 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:37:54 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200456 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:37:54 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200462 | controller |       | ++ grep watcher\n    logger.go:42: 09:37:54 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200468 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.200474 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:37:56 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200480 | controller |       | + SERVICEID=e0a908c724554093bb8de86515131a2c\n    logger.go:42: 09:37:56 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200486 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.200492 | controller |       \   logger.go:42: 09:37:57 | watcher-tls/3-disable-podlevel-tls | + '[' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.200498 | controller |       == e0a908c724554093bb8de86515131a2c ']'\n    logger.go:42: 09:37:57 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200504 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:44:27.200510 | controller |       \   logger.go:42: 09:37:57 | watcher-tls/3-disable-podlevel-tls | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.200520 | controller |       ']'\n    logger.go:42: 09:37:57 | watcher-tls/3-disable-podlevel-tls | + oc exec
2026-03-14 09:44:27.200526 | controller |       -n watcher-kuttl-default openstackclient -- openstack endpoint list\n    logger.go:42:
2026-03-14 09:44:27.200533 | controller |       09:37:57 | watcher-tls/3-disable-podlevel-tls | + grep internal\n    logger.go:42:
2026-03-14 09:44:27.200539 | controller |       09:37:57 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n    logger.go:42:
2026-03-14 09:44:27.200545 | controller |       09:37:57 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n    logger.go:42:
2026-03-14 09:44:27.200554 | controller |       09:37:59 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 0 ']'\n    logger.go:42:
2026-03-14 09:44:27.200560 | controller |       09:38:00 | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo
2026-03-14 09:44:27.200566 | controller |       pipefail\n        oc exec -n ${NAMESPACE} openstackclient -- openstack service list
2026-03-14 09:44:27.200572 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc exec
2026-03-14 09:44:27.200578 | controller |       -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name -c Type
2026-03-14 09:44:27.200584 | controller |       -c ID | grep watcher| awk '{print $1}')\n        [ $(oc get -n ${NAMESPACE} keystoneservice
2026-03-14 09:44:27.200590 | controller |       watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [ -n \"$(oc
2026-03-14 09:44:27.200596 | controller |       get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n
2026-03-14 09:44:27.200603 | controller |       \       # check that watcher internal endpoint does not use https\n        oc exec
2026-03-14 09:44:27.200609 | controller |       -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim |
2026-03-14 09:44:27.200615 | controller |       grep internal  | [ $(grep -c https) == 0 ]\n        # check that watcher public
2026-03-14 09:44:27.200621 | controller |       endpoint does use https\n        oc exec -n ${NAMESPACE} openstackclient -- openstack
2026-03-14 09:44:27.200627 | controller |       endpoint list | grep infra-optim | grep public  | [ $(grep -c https) == 1 ]\n        #
2026-03-14 09:44:27.200633 | controller |       If we are running the container locally, skip following test\n        if [ \"$(oc
2026-03-14 09:44:27.200639 | controller |       get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:44:27.200644 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:44:27.200650 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:44:27.200659 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:44:27.200665 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:44:27.200671 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:44:27.200676 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:44:27.200682 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:44:27.200688 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:38:00 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200693 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.200700 | controller |       value -c Name -c Type\n    logger.go:42: 09:38:00 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200706 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:38:02 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200722 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:38:02 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200730 | controller |       | ++ grep watcher\n    logger.go:42: 09:38:02 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200736 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:38:02 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200742 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.200748 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:38:04 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200754 | controller |       | + SERVICEID=e0a908c724554093bb8de86515131a2c\n    logger.go:42: 09:38:04 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200760 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.200766 | controller |       \   logger.go:42: 09:38:05 | watcher-tls/3-disable-podlevel-tls | + '[' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.200776 | controller |       == e0a908c724554093bb8de86515131a2c ']'\n    logger.go:42: 09:38:05 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200783 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:44:27.200789 | controller |       \   logger.go:42: 09:38:05 | watcher-tls/3-disable-podlevel-tls | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.200795 | controller |       ']'\n    logger.go:42: 09:38:05 | watcher-tls/3-disable-podlevel-tls | + grep internal\n
2026-03-14 09:44:27.200802 | controller |       \   logger.go:42: 09:38:05 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n
2026-03-14 09:44:27.200808 | controller |       \   logger.go:42: 09:38:05 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.200814 | controller |       openstackclient -- openstack endpoint list\n    logger.go:42: 09:38:05 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200820 | controller |       | ++ grep -c https\n    logger.go:42: 09:38:07 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200826 | controller |       | + '[' 1 == 0 ']'\n    logger.go:42: 09:38:08 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200832 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:44:27.200838 | controller |       -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:44:27.200844 | controller |       \       SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.200851 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n        [
2026-03-14 09:44:27.200857 | controller |       $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.200863 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl
2026-03-14 09:44:27.200870 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # check that watcher internal endpoint
2026-03-14 09:44:27.200876 | controller |       does not use https\n        oc exec -n ${NAMESPACE} openstackclient -- openstack
2026-03-14 09:44:27.200883 | controller |       endpoint list | grep infra-optim | grep internal  | [ $(grep -c https) == 0 ]\n
2026-03-14 09:44:27.200890 | controller |       \       # check that watcher public endpoint does use https\n        oc exec -n
2026-03-14 09:44:27.200896 | controller |       ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep
2026-03-14 09:44:27.200903 | controller |       public  | [ $(grep -c https) == 1 ]\n        # If we are running the container locally,
2026-03-14 09:44:27.200910 | controller |       skip following test\n        if [ \"$(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.200917 | controller |       -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit 0\n
2026-03-14 09:44:27.200924 | controller |       \       fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:44:27.200930 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:44:27.200937 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:44:27.200944 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:44:27.200950 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:44:27.200956 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:44:27.200966 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:44:27.200973 | controller |       \       ]\n    logger.go:42: 09:38:08 | watcher-tls/3-disable-podlevel-tls | + oc
2026-03-14 09:44:27.200979 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.200985 | controller |       -c Name -c Type\n    logger.go:42: 09:38:08 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.200993 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:38:10 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201008 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:38:10 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201015 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:38:10 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201021 | controller |       | ++ grep watcher\n    logger.go:42: 09:38:10 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201028 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.201034 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:38:12 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201040 | controller |       | + SERVICEID=e0a908c724554093bb8de86515131a2c\n    logger.go:42: 09:38:12 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201046 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.201052 | controller |       \   logger.go:42: 09:38:12 | watcher-tls/3-disable-podlevel-tls | + '[' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.201058 | controller |       == e0a908c724554093bb8de86515131a2c ']'\n    logger.go:42: 09:38:12 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201065 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:44:27.201071 | controller |       \   logger.go:42: 09:38:12 | watcher-tls/3-disable-podlevel-tls | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.201077 | controller |       ']'\n    logger.go:42: 09:38:12 | watcher-tls/3-disable-podlevel-tls | + grep internal\n
2026-03-14 09:44:27.201083 | controller |       \   logger.go:42: 09:38:12 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n
2026-03-14 09:44:27.201089 | controller |       \   logger.go:42: 09:38:12 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.201096 | controller |       openstackclient -- openstack endpoint list\n    logger.go:42: 09:38:12 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201102 | controller |       | ++ grep -c https\n    logger.go:42: 09:38:14 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201108 | controller |       | + '[' 1 == 0 ']'\n    logger.go:42: 09:38:15 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201117 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:44:27.201123 | controller |       -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:44:27.201130 | controller |       \       SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.201136 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n        [
2026-03-14 09:44:27.201147 | controller |       $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.201154 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl
2026-03-14 09:44:27.201160 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # check that watcher internal endpoint
2026-03-14 09:44:27.201166 | controller |       does not use https\n        oc exec -n ${NAMESPACE} openstackclient -- openstack
2026-03-14 09:44:27.201172 | controller |       endpoint list | grep infra-optim | grep internal  | [ $(grep -c https) == 0 ]\n
2026-03-14 09:44:27.201179 | controller |       \       # check that watcher public endpoint does use https\n        oc exec -n
2026-03-14 09:44:27.201185 | controller |       ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep
2026-03-14 09:44:27.201192 | controller |       public  | [ $(grep -c https) == 1 ]\n        # If we are running the container locally,
2026-03-14 09:44:27.201198 | controller |       skip following test\n        if [ \"$(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.201205 | controller |       -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit 0\n
2026-03-14 09:44:27.201214 | controller |       \       fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:44:27.201220 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:44:27.201227 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:44:27.201233 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:44:27.201240 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:44:27.201246 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:44:27.201252 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:44:27.201260 | controller |       \       ]\n    logger.go:42: 09:38:15 | watcher-tls/3-disable-podlevel-tls | + oc
2026-03-14 09:44:27.201267 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.201273 | controller |       -c Name -c Type\n    logger.go:42: 09:38:15 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201280 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:38:17 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201286 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:38:17 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201323 | controller |       | ++ grep watcher\n    logger.go:42: 09:38:17 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201330 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.201337 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:38:17 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201343 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:38:19 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201349 | controller |       | + SERVICEID=e0a908c724554093bb8de86515131a2c\n    logger.go:42: 09:38:19 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201356 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.201362 | controller |       \   logger.go:42: 09:38:19 | watcher-tls/3-disable-podlevel-tls | + '[' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.201368 | controller |       == e0a908c724554093bb8de86515131a2c ']'\n    logger.go:42: 09:38:19 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201375 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:44:27.201381 | controller |       \   logger.go:42: 09:38:19 | watcher-tls/3-disable-podlevel-tls | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.201388 | controller |       ']'\n    logger.go:42: 09:38:19 | watcher-tls/3-disable-podlevel-tls | + grep internal\n
2026-03-14 09:44:27.201394 | controller |       \   logger.go:42: 09:38:19 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n
2026-03-14 09:44:27.201400 | controller |       \   logger.go:42: 09:38:19 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.201407 | controller |       openstackclient -- openstack endpoint list\n    logger.go:42: 09:38:19 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201413 | controller |       | ++ grep -c https\n    logger.go:42: 09:38:21 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201419 | controller |       | + '[' 1 == 0 ']'\n    logger.go:42: 09:38:22 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201426 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:44:27.201432 | controller |       -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:44:27.201439 | controller |       \       SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.201445 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n        [
2026-03-14 09:44:27.201454 | controller |       $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.201461 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl
2026-03-14 09:44:27.201467 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # check that watcher internal endpoint
2026-03-14 09:44:27.201474 | controller |       does not use https\n        oc exec -n ${NAMESPACE} openstackclient -- openstack
2026-03-14 09:44:27.201480 | controller |       endpoint list | grep infra-optim | grep internal  | [ $(grep -c https) == 0 ]\n
2026-03-14 09:44:27.201486 | controller |       \       # check that watcher public endpoint does use https\n        oc exec -n
2026-03-14 09:44:27.201492 | controller |       ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep
2026-03-14 09:44:27.201499 | controller |       public  | [ $(grep -c https) == 1 ]\n        # If we are running the container locally,
2026-03-14 09:44:27.201505 | controller |       skip following test\n        if [ \"$(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.201511 | controller |       -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit 0\n
2026-03-14 09:44:27.201518 | controller |       \       fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:44:27.201524 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:44:27.201530 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:44:27.201537 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:44:27.201543 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:44:27.201549 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:44:27.201555 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:44:27.201562 | controller |       \       ]\n    logger.go:42: 09:38:22 | watcher-tls/3-disable-podlevel-tls | + oc
2026-03-14 09:44:27.201571 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.201577 | controller |       -c Name -c Type\n    logger.go:42: 09:38:22 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201584 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:38:26 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201590 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:38:26 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201604 | controller |       | ++ grep watcher\n    logger.go:42: 09:38:26 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201611 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:38:26 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201617 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.201624 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:38:29 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201629 | controller |       | + SERVICEID=e0a908c724554093bb8de86515131a2c\n    logger.go:42: 09:38:29 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201635 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.201642 | controller |       \   logger.go:42: 09:38:30 | watcher-tls/3-disable-podlevel-tls | + '[' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.201648 | controller |       == e0a908c724554093bb8de86515131a2c ']'\n    logger.go:42: 09:38:30 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201654 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:44:27.201661 | controller |       \   logger.go:42: 09:38:31 | watcher-tls/3-disable-podlevel-tls | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.201671 | controller |       ']'\n    logger.go:42: 09:38:31 | watcher-tls/3-disable-podlevel-tls | + grep internal\n
2026-03-14 09:44:27.201678 | controller |       \   logger.go:42: 09:38:31 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n
2026-03-14 09:44:27.201707 | controller |       \   logger.go:42: 09:38:31 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n
2026-03-14 09:44:27.201714 | controller |       \   logger.go:42: 09:38:31 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.201720 | controller |       openstackclient -- openstack endpoint list\n    logger.go:42: 09:38:33 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201726 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:38:33 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201732 | controller |       | + grep public\n    logger.go:42: 09:38:33 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201739 | controller |       | + grep infra-optim\n    logger.go:42: 09:38:33 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201745 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list\n
2026-03-14 09:44:27.201751 | controller |       \   logger.go:42: 09:38:33 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n
2026-03-14 09:44:27.201758 | controller |       \   logger.go:42: 09:38:35 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 1 ']'\n
2026-03-14 09:44:27.201764 | controller |       \   logger.go:42: 09:38:35 | watcher-tls/3-disable-podlevel-tls | ++ oc get pods
2026-03-14 09:44:27.201770 | controller |       -n openstack-operators -o name -l openstack.org/operator-name=watcher\n    logger.go:42:
2026-03-14 09:44:27.201776 | controller |       09:38:35 | watcher-tls/3-disable-podlevel-tls | + '[' '' == '' ']'\n    logger.go:42:
2026-03-14 09:44:27.201783 | controller |       09:38:35 | watcher-tls/3-disable-podlevel-tls | + exit 0\n    logger.go:42: 09:38:36
2026-03-14 09:44:27.201789 | controller |       | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:44:27.201795 | controller |       \       oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.201801 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n ${NAMESPACE}
2026-03-14 09:44:27.201808 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep
2026-03-14 09:44:27.201814 | controller |       watcher| awk '{print $1}')\n        [ $(oc get -n ${NAMESPACE} keystoneservice watcher
2026-03-14 09:44:27.201820 | controller |       -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE}
2026-03-14 09:44:27.201827 | controller |       watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n        # check that
2026-03-14 09:44:27.201833 | controller |       watcher internal endpoint does not use https\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:44:27.201839 | controller |       -- openstack endpoint list | grep infra-optim | grep internal  | [ $(grep -c https)
2026-03-14 09:44:27.201846 | controller |       == 0 ]\n        # check that watcher public endpoint does use https\n        oc
2026-03-14 09:44:27.201852 | controller |       exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim
2026-03-14 09:44:27.201858 | controller |       | grep public  | [ $(grep -c https) == 1 ]\n        # If we are running the container
2026-03-14 09:44:27.201864 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:44:27.201871 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:44:27.201880 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:44:27.201887 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:44:27.201893 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:44:27.201899 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:44:27.201914 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:44:27.201923 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:44:27.201929 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:44:27.201936 | controller |       \       ]\n    logger.go:42: 09:38:36 | watcher-tls/3-disable-podlevel-tls | + oc
2026-03-14 09:44:27.201942 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.201949 | controller |       -c Name -c Type\n    logger.go:42: 09:38:36 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201957 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:38:38 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201963 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:38:38 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201971 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.201978 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:38:38 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201984 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:38:38 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201991 | controller |       | ++ grep watcher\n    logger.go:42: 09:38:40 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.201997 | controller |       | + SERVICEID=e0a908c724554093bb8de86515131a2c\n    logger.go:42: 09:38:40 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.202003 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.202009 | controller |       \   logger.go:42: 09:38:40 | watcher-tls/3-disable-podlevel-tls | + '[' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.202015 | controller |       == e0a908c724554093bb8de86515131a2c ']'\n    logger.go:42: 09:38:40 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.202021 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:44:27.202027 | controller |       \   logger.go:42: 09:38:40 | watcher-tls/3-disable-podlevel-tls | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.202033 | controller |       ']'\n    logger.go:42: 09:38:40 | watcher-tls/3-disable-podlevel-tls | + grep internal\n
2026-03-14 09:44:27.202039 | controller |       \   logger.go:42: 09:38:40 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n
2026-03-14 09:44:27.202045 | controller |       \   logger.go:42: 09:38:40 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n
2026-03-14 09:44:27.202054 | controller |       \   logger.go:42: 09:38:40 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.202061 | controller |       openstackclient -- openstack endpoint list\n    logger.go:42: 09:38:42 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.202067 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:38:42 | watcher-tls/3-disable-podlevel-tls
2026-03-14 09:44:27.202073 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list\n
2026-03-14 09:44:27.202079 | controller |       \   logger.go:42: 09:38:42 | watcher-tls/3-disable-podlevel-tls | + grep public\n
2026-03-14 09:44:27.202085 | controller |       \   logger.go:42: 09:38:42 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n
2026-03-14 09:44:27.202092 | controller |       \   logger.go:42: 09:38:42 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n
2026-03-14 09:44:27.202097 | controller |       \   logger.go:42: 09:38:45 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 1 ']'\n
2026-03-14 09:44:27.202103 | controller |       \   logger.go:42: 09:38:45 | watcher-tls/3-disable-podlevel-tls | ++ oc get pods
2026-03-14 09:44:27.202109 | controller |       -n openstack-operators -o name -l openstack.org/operator-name=watcher\n    logger.go:42:
2026-03-14 09:44:27.202116 | controller |       09:38:45 | watcher-tls/3-disable-podlevel-tls | + '[' '' == '' ']'\n    logger.go:42:
2026-03-14 09:44:27.202122 | controller |       09:38:45 | watcher-tls/3-disable-podlevel-tls | + exit 0\n    logger.go:42: 09:38:45
2026-03-14 09:44:27.202132 | controller |       | watcher-tls/3-disable-podlevel-tls | test step completed 3-disable-podlevel-tls\n
2026-03-14 09:44:27.202138 | controller |       \   logger.go:42: 09:38:45 | watcher-tls/4-deploy-without-route | starting test
2026-03-14 09:44:27.202144 | controller |       step 4-deploy-without-route\n    logger.go:42: 09:38:45 | watcher-tls/4-deploy-without-route
2026-03-14 09:44:27.202150 | controller |       | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json'
2026-03-14 09:44:27.202157 | controller |       -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/override\", \"value\":{\"service\":
2026-03-14 09:44:27.202163 | controller |       { \"internal\": {}, \"public\": { \"metadata\": { \"annotations\": { \"metallb.universe.tf/address-pool\":
2026-03-14 09:44:27.202169 | controller |       \"ctlplane\", \"metallb.universe.tf/allow-shared-ip\": \"ctlplane\" } }, \"spec\":
2026-03-14 09:44:27.202175 | controller |       { \"type\": \"LoadBalancer\" } } } }}]'\n        ]\n    logger.go:42: 09:38:45 |
2026-03-14 09:44:27.202184 | controller |       watcher-tls/4-deploy-without-route | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:44:27.202199 | controller |       patched\n    logger.go:42: 09:38:45 | watcher-tls/4-deploy-without-route | running
2026-03-14 09:44:27.202205 | controller |       command: [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:44:27.202212 | controller |       -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:44:27.202218 | controller |       \       SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.202224 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n        [
2026-03-14 09:44:27.202230 | controller |       $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.202236 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl
2026-03-14 09:44:27.202242 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # If we are running the container
2026-03-14 09:44:27.202248 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:44:27.202254 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:44:27.202261 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:44:27.202267 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:44:27.202273 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:44:27.202279 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:44:27.202285 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:44:27.202311 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:44:27.202318 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:44:27.202324 | controller |       \       ]\n    logger.go:42: 09:38:45 | watcher-tls/4-deploy-without-route | + oc
2026-03-14 09:44:27.202330 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.202336 | controller |       -c Name -c Type\n    logger.go:42: 09:38:45 | watcher-tls/4-deploy-without-route
2026-03-14 09:44:27.202342 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:38:47 | watcher-tls/4-deploy-without-route
2026-03-14 09:44:27.202348 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:38:47 | watcher-tls/4-deploy-without-route
2026-03-14 09:44:27.202354 | controller |       | ++ grep watcher\n    logger.go:42: 09:38:47 | watcher-tls/4-deploy-without-route
2026-03-14 09:44:27.202361 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:38:47 | watcher-tls/4-deploy-without-route
2026-03-14 09:44:27.202367 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.202376 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:38:49 | watcher-tls/4-deploy-without-route
2026-03-14 09:44:27.202383 | controller |       | + SERVICEID=e0a908c724554093bb8de86515131a2c\n    logger.go:42: 09:38:49 | watcher-tls/4-deploy-without-route
2026-03-14 09:44:27.202389 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.202395 | controller |       \   logger.go:42: 09:38:49 | watcher-tls/4-deploy-without-route | + '[' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.202401 | controller |       == e0a908c724554093bb8de86515131a2c ']'\n    logger.go:42: 09:38:49 | watcher-tls/4-deploy-without-route
2026-03-14 09:44:27.202407 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:44:27.202413 | controller |       \   logger.go:42: 09:38:50 | watcher-tls/4-deploy-without-route | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.202419 | controller |       ']'\n    logger.go:42: 09:38:50 | watcher-tls/4-deploy-without-route | ++ oc get
2026-03-14 09:44:27.202428 | controller |       pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n    logger.go:42:
2026-03-14 09:44:27.202434 | controller |       09:38:50 | watcher-tls/4-deploy-without-route | + '[' '' == '' ']'\n    logger.go:42:
2026-03-14 09:44:27.202440 | controller |       09:38:50 | watcher-tls/4-deploy-without-route | + exit 0\n    logger.go:42: 09:38:51
2026-03-14 09:44:27.202446 | controller |       | watcher-tls/4-deploy-without-route | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:44:27.202452 | controller |       \       oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.202458 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n ${NAMESPACE}
2026-03-14 09:44:27.202464 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep
2026-03-14 09:44:27.202470 | controller |       watcher| awk '{print $1}')\n        [ $(oc get -n ${NAMESPACE} keystoneservice watcher
2026-03-14 09:44:27.202477 | controller |       -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE}
2026-03-14 09:44:27.202492 | controller |       watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n        # If we are
2026-03-14 09:44:27.202503 | controller |       running the container locally, skip following test\n        if [ \"$(oc get pods
2026-03-14 09:44:27.202509 | controller |       -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\"
2026-03-14 09:44:27.202516 | controller |       ]; then\n            exit 0\n        fi\n        env_variables=$(oc set env $(oc
2026-03-14 09:44:27.202522 | controller |       get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:44:27.202528 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:44:27.202534 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:44:27.202540 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:44:27.202547 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:44:27.202553 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:44:27.202559 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:38:51 | watcher-tls/4-deploy-without-route
2026-03-14 09:44:27.202566 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.202572 | controller |       value -c Name -c Type\n    logger.go:42: 09:38:51 | watcher-tls/4-deploy-without-route
2026-03-14 09:44:27.202578 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:38:53 | watcher-tls/4-deploy-without-route
2026-03-14 09:44:27.202584 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:38:53 | watcher-tls/4-deploy-without-route
2026-03-14 09:44:27.202593 | controller |       | ++ grep watcher\n    logger.go:42: 09:38:53 | watcher-tls/4-deploy-without-route
2026-03-14 09:44:27.202600 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:38:53 | watcher-tls/4-deploy-without-route
2026-03-14 09:44:27.202606 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.202613 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:38:55 | watcher-tls/4-deploy-without-route
2026-03-14 09:44:27.202619 | controller |       | + SERVICEID=e0a908c724554093bb8de86515131a2c\n    logger.go:42: 09:38:55 | watcher-tls/4-deploy-without-route
2026-03-14 09:44:27.202625 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.202631 | controller |       \   logger.go:42: 09:38:55 | watcher-tls/4-deploy-without-route | + '[' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.202637 | controller |       == e0a908c724554093bb8de86515131a2c ']'\n    logger.go:42: 09:38:55 | watcher-tls/4-deploy-without-route
2026-03-14 09:44:27.202643 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:44:27.202650 | controller |       \   logger.go:42: 09:38:55 | watcher-tls/4-deploy-without-route | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.202657 | controller |       ']'\n    logger.go:42: 09:38:55 | watcher-tls/4-deploy-without-route | ++ oc get
2026-03-14 09:44:27.202663 | controller |       pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n    logger.go:42:
2026-03-14 09:44:27.202669 | controller |       09:38:55 | watcher-tls/4-deploy-without-route | + '[' '' == '' ']'\n    logger.go:42:
2026-03-14 09:44:27.202676 | controller |       09:38:55 | watcher-tls/4-deploy-without-route | + exit 0\n    logger.go:42: 09:38:55
2026-03-14 09:44:27.202682 | controller |       | watcher-tls/4-deploy-without-route | test step completed 4-deploy-without-route\n
2026-03-14 09:44:27.202688 | controller |       \   logger.go:42: 09:38:55 | watcher-tls/5-disable-tls | starting test step 5-disable-tls\n
2026-03-14 09:44:27.202694 | controller |       \   logger.go:42: 09:38:55 | watcher-tls/5-disable-tls | running command: [sh -c
2026-03-14 09:44:27.202701 | controller |       oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' -p='[{\"op\": \"replace\",
2026-03-14 09:44:27.202707 | controller |       \"path\": \"/spec/apiServiceTemplate/override\", \"value\":{}}]'\n        ]\n    logger.go:42:
2026-03-14 09:44:27.202713 | controller |       09:38:55 | watcher-tls/5-disable-tls | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:44:27.202719 | controller |       patched\n    logger.go:42: 09:38:55 | watcher-tls/5-disable-tls | running command:
2026-03-14 09:44:27.202726 | controller |       [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient -- openstack
2026-03-14 09:44:27.202732 | controller |       service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc
2026-03-14 09:44:27.202738 | controller |       exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.202745 | controller |       -c Type -c ID | grep watcher| awk '{print $1}')\n        [ $(oc get -n ${NAMESPACE}
2026-03-14 09:44:27.202751 | controller |       keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [
2026-03-14 09:44:27.202757 | controller |       -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\"
2026-03-14 09:44:27.202764 | controller |       ]\n        # If we are running the container locally, skip following test\n        if
2026-03-14 09:44:27.202780 | controller |       [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:44:27.202787 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:44:27.202794 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:44:27.202803 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:44:27.202810 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:44:27.202819 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:44:27.202825 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:44:27.202831 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:44:27.202837 | controller |       variables found.\"\n        fi\n        # check that no watcher endpoint uses https\n
2026-03-14 09:44:27.202843 | controller |       \       oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.202848 | controller |       infra-optim | [ $(grep -c https) == 0 ]\n        ]\n    logger.go:42: 09:38:55 |
2026-03-14 09:44:27.202854 | controller |       watcher-tls/5-disable-tls | + oc exec -n watcher-kuttl-default openstackclient --
2026-03-14 09:44:27.202860 | controller |       openstack service list -f value -c Name -c Type\n    logger.go:42: 09:38:55 | watcher-tls/5-disable-tls
2026-03-14 09:44:27.202866 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:38:57 | watcher-tls/5-disable-tls
2026-03-14 09:44:27.202873 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:38:57 | watcher-tls/5-disable-tls | ++
2026-03-14 09:44:27.202879 | controller |       grep watcher\n    logger.go:42: 09:38:57 | watcher-tls/5-disable-tls | ++ awk '{print
2026-03-14 09:44:27.202888 | controller |       $1}'\n    logger.go:42: 09:38:57 | watcher-tls/5-disable-tls | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.202895 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID\n    logger.go:42:
2026-03-14 09:44:27.202901 | controller |       09:38:59 | watcher-tls/5-disable-tls | + SERVICEID=e0a908c724554093bb8de86515131a2c\n
2026-03-14 09:44:27.202908 | controller |       \   logger.go:42: 09:38:59 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.202914 | controller |       keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n    logger.go:42: 09:38:59
2026-03-14 09:44:27.202920 | controller |       | watcher-tls/5-disable-tls | + '[' e0a908c724554093bb8de86515131a2c == e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.202927 | controller |       ']'\n    logger.go:42: 09:38:59 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.202933 | controller |       watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n    logger.go:42: 09:39:00
2026-03-14 09:44:27.202940 | controller |       | watcher-tls/5-disable-tls | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.202946 | controller |       ']'\n    logger.go:42: 09:39:00 | watcher-tls/5-disable-tls | ++ oc get pods -n
2026-03-14 09:44:27.202953 | controller |       openstack-operators -o name -l openstack.org/operator-name=watcher\n    logger.go:42:
2026-03-14 09:44:27.202959 | controller |       09:39:00 | watcher-tls/5-disable-tls | + '[' '' == '' ']'\n    logger.go:42: 09:39:00
2026-03-14 09:44:27.202966 | controller |       | watcher-tls/5-disable-tls | + exit 0\n    logger.go:42: 09:39:01 | watcher-tls/5-disable-tls
2026-03-14 09:44:27.202972 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n ${NAMESPACE} openstackclient
2026-03-14 09:44:27.202979 | controller |       -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:44:27.202985 | controller |       \       SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.202991 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n        [
2026-03-14 09:44:27.202998 | controller |       $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.203004 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl
2026-03-14 09:44:27.203011 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # If we are running the container
2026-03-14 09:44:27.203017 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:44:27.203026 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:44:27.203032 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:44:27.203039 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:44:27.203046 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:44:27.203052 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:44:27.203059 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:44:27.203074 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:44:27.203082 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:44:27.203089 | controller |       \       # check that no watcher endpoint uses https\n        oc exec -n ${NAMESPACE}
2026-03-14 09:44:27.203096 | controller |       openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https)
2026-03-14 09:44:27.203105 | controller |       == 0 ]\n        ]\n    logger.go:42: 09:39:01 | watcher-tls/5-disable-tls | + oc
2026-03-14 09:44:27.203112 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.203119 | controller |       -c Name -c Type\n    logger.go:42: 09:39:01 | watcher-tls/5-disable-tls | ++ grep
2026-03-14 09:44:27.203125 | controller |       -c '^watcher'\n    logger.go:42: 09:39:03 | watcher-tls/5-disable-tls | + '[' 1
2026-03-14 09:44:27.203132 | controller |       == 1 ']'\n    logger.go:42: 09:39:03 | watcher-tls/5-disable-tls | ++ oc exec -n
2026-03-14 09:44:27.203139 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.203145 | controller |       -c Type -c ID\n    logger.go:42: 09:39:03 | watcher-tls/5-disable-tls | ++ awk '{print
2026-03-14 09:44:27.203152 | controller |       $1}'\n    logger.go:42: 09:39:03 | watcher-tls/5-disable-tls | ++ grep watcher\n
2026-03-14 09:44:27.203158 | controller |       \   logger.go:42: 09:39:05 | watcher-tls/5-disable-tls | + SERVICEID=e0a908c724554093bb8de86515131a2c\n
2026-03-14 09:44:27.203165 | controller |       \   logger.go:42: 09:39:05 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.203172 | controller |       keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n    logger.go:42: 09:39:05
2026-03-14 09:44:27.203178 | controller |       | watcher-tls/5-disable-tls | + '[' e0a908c724554093bb8de86515131a2c == e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.203184 | controller |       ']'\n    logger.go:42: 09:39:05 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.203191 | controller |       watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n    logger.go:42: 09:39:05
2026-03-14 09:44:27.203197 | controller |       | watcher-tls/5-disable-tls | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.203204 | controller |       ']'\n    logger.go:42: 09:39:05 | watcher-tls/5-disable-tls | ++ oc get pods -n
2026-03-14 09:44:27.203211 | controller |       openstack-operators -o name -l openstack.org/operator-name=watcher\n    logger.go:42:
2026-03-14 09:44:27.203218 | controller |       09:39:05 | watcher-tls/5-disable-tls | + '[' '' == '' ']'\n    logger.go:42: 09:39:05
2026-03-14 09:44:27.203224 | controller |       | watcher-tls/5-disable-tls | + exit 0\n    logger.go:42: 09:39:05 | watcher-tls/5-disable-tls
2026-03-14 09:44:27.203230 | controller |       | test step completed 5-disable-tls\n    logger.go:42: 09:39:05 | watcher-tls/6-cleanup-watcher
2026-03-14 09:44:27.203237 | controller |       | starting test step 6-cleanup-watcher\n    logger.go:42: 09:39:05 | watcher-tls/6-cleanup-watcher
2026-03-14 09:44:27.203244 | controller |       | running command: [sh -c set -ex\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.203250 | controller |       -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0
2026-03-14 09:44:27.203261 | controller |       ]\n        ]\n    logger.go:42: 09:39:05 | watcher-tls/6-cleanup-watcher | + oc
2026-03-14 09:44:27.203269 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.203275 | controller |       -c Name -c Type\n    logger.go:42: 09:39:05 | watcher-tls/6-cleanup-watcher | ++
2026-03-14 09:44:27.203282 | controller |       grep -c '^watcher'\n    logger.go:42: 09:39:08 | watcher-tls/6-cleanup-watcher |
2026-03-14 09:44:27.203305 | controller |       + '[' 1 == 0 ']'\n    logger.go:42: 09:39:09 | watcher-tls/6-cleanup-watcher | running
2026-03-14 09:44:27.203316 | controller |       command: [sh -c set -ex\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.203323 | controller |       -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0
2026-03-14 09:44:27.203329 | controller |       ]\n        ]\n    logger.go:42: 09:39:09 | watcher-tls/6-cleanup-watcher | + oc
2026-03-14 09:44:27.203336 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.203342 | controller |       -c Name -c Type\n    logger.go:42: 09:39:09 | watcher-tls/6-cleanup-watcher | ++
2026-03-14 09:44:27.203349 | controller |       grep -c '^watcher'\n    logger.go:42: 09:39:11 | watcher-tls/6-cleanup-watcher |
2026-03-14 09:44:27.203355 | controller |       + '[' 0 == 0 ']'\n    logger.go:42: 09:39:12 | watcher-tls/6-cleanup-watcher | running
2026-03-14 09:44:27.203362 | controller |       command: [sh -c set -ex\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.203368 | controller |       -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0
2026-03-14 09:44:27.203375 | controller |       ]\n        ]\n    logger.go:42: 09:39:12 | watcher-tls/6-cleanup-watcher | + oc
2026-03-14 09:44:27.203382 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.203388 | controller |       -c Name -c Type\n    logger.go:42: 09:39:12 | watcher-tls/6-cleanup-watcher | ++
2026-03-14 09:44:27.203407 | controller |       grep -c '^watcher'\n    logger.go:42: 09:39:14 | watcher-tls/6-cleanup-watcher |
2026-03-14 09:44:27.203415 | controller |       + '[' 0 == 0 ']'\n    logger.go:42: 09:39:14 | watcher-tls/6-cleanup-watcher | test
2026-03-14 09:44:27.203422 | controller |       step completed 6-cleanup-watcher\n    logger.go:42: 09:39:14 | watcher-tls/7-cleanup-certs
2026-03-14 09:44:27.203429 | controller |       | starting test step 7-cleanup-certs\n    logger.go:42: 09:39:14 | watcher-tls/7-cleanup-certs
2026-03-14 09:44:27.203435 | controller |       | test step completed 7-cleanup-certs\n    logger.go:42: 09:39:14 | watcher-tls
2026-03-14 09:44:27.203442 | controller |       | skipping kubernetes event logging\n=== CONT  kuttl/harness/watcher-rmquser\n    logger.go:42:
2026-03-14 09:44:27.203448 | controller |       09:39:14 | watcher-rmquser | Skipping creation of user-supplied namespace: watcher-kuttl-default\n
2026-03-14 09:44:27.203455 | controller |       \   logger.go:42: 09:39:14 | watcher-rmquser/0-cleanup-watcher | starting test step
2026-03-14 09:44:27.203461 | controller |       0-cleanup-watcher\n    logger.go:42: 09:39:14 | watcher-rmquser/0-cleanup-watcher
2026-03-14 09:44:27.203468 | controller |       | test step completed 0-cleanup-watcher\n    logger.go:42: 09:39:14 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.203474 | controller |       | starting test step 1-deploy\n    logger.go:42: 09:39:14 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.203481 | controller |       | Watcher:watcher-kuttl-default/watcher-kuttl created\n    logger.go:42: 09:39:14
2026-03-14 09:44:27.203488 | controller |       | watcher-rmquser/1-deploy | running command: [sh -c set -euxo pipefail\n        \n
2026-03-14 09:44:27.203494 | controller |       \       # Wait for Watcher to be Ready\n        kubectl wait --for=condition=Ready
2026-03-14 09:44:27.203501 | controller |       watcher/watcher-kuttl -n $NAMESPACE --timeout=300s\n        \n        # Verify WatcherNotificationTransportURLReady
2026-03-14 09:44:27.203508 | controller |       condition exists and is True\n        kubectl get watcher watcher-kuttl -n $NAMESPACE
2026-03-14 09:44:27.203514 | controller |       -o jsonpath='{.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}'
2026-03-14 09:44:27.203523 | controller |       | grep -q \"True\"\n        echo \"WatcherNotificationTransportURLReady condition
2026-03-14 09:44:27.203529 | controller |       is True\"\n        \n        # Count TransportURL CRs - should be exactly 2 (one
2026-03-14 09:44:27.203536 | controller |       for messaging, one for notifications)\n        transport_count=$(kubectl get transporturl
2026-03-14 09:44:27.203542 | controller |       -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-transport\" | wc -l)\n        notification_transport_count=$(kubectl
2026-03-14 09:44:27.203549 | controller |       get transporturl -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-notification\"
2026-03-14 09:44:27.203555 | controller |       | wc -l)\n        \n        if [ \"$transport_count\" -ne \"1\" ]; then\n          echo
2026-03-14 09:44:27.203562 | controller |       \"Expected 1 watcher-transport TransportURL, found $transport_count\"\n          exit
2026-03-14 09:44:27.203568 | controller |       1\n        fi\n        \n        if [ \"$notification_transport_count\" -ne \"1\"
2026-03-14 09:44:27.203575 | controller |       ]; then\n          echo \"Expected 1 notification-transport TransportURL, found
2026-03-14 09:44:27.203582 | controller |       $notification_transport_count\"\n          exit 1\n        fi\n        \n        echo
2026-03-14 09:44:27.203588 | controller |       \"Correctly found 2 TransportURLs (separate clusters: transport and notification)\"\n
2026-03-14 09:44:27.203595 | controller |       \       \n        # Verify watcher-transport has correct user and vhost\n        transport_user=$(kubectl
2026-03-14 09:44:27.203602 | controller |       get transporturl watcher-kuttl-watcher-transport -n $NAMESPACE -o jsonpath='{.spec.username}')\n
2026-03-14 09:44:27.203608 | controller |       \       transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport
2026-03-14 09:44:27.203615 | controller |       -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n        if [ \"$transport_user\" !=
2026-03-14 09:44:27.203621 | controller |       \"watcher-rpc\" ]; then\n          echo \"Expected watcher-transport username 'watcher-rpc',
2026-03-14 09:44:27.203628 | controller |       found '$transport_user'\"\n          exit 1\n        fi\n        if [ \"$transport_vhost\"
2026-03-14 09:44:27.203634 | controller |       != \"watcher-rpc\" ]; then\n          echo \"Expected watcher-transport vhost 'watcher-rpc',
2026-03-14 09:44:27.203640 | controller |       found '$transport_vhost'\"\n          exit 1\n        fi\n        echo \"Watcher
2026-03-14 09:44:27.203647 | controller |       transport has correct user (watcher-rpc) and vhost (watcher-rpc)\"\n        \n        #
2026-03-14 09:44:27.203654 | controller |       Verify notification-transport has correct user and vhost\n        notif_user=$(kubectl
2026-03-14 09:44:27.203660 | controller |       get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE
2026-03-14 09:44:27.203667 | controller |       -o jsonpath='{.spec.username}')\n        notif_vhost=$(kubectl get transporturl
2026-03-14 09:44:27.203677 | controller |       watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n
2026-03-14 09:44:27.203684 | controller |       \       if [ \"$notif_user\" != \"watcher-notifications\" ]; then\n          echo
2026-03-14 09:44:27.203691 | controller |       \"Expected notification-transport username 'watcher-notifications', found '$notif_user'\"\n
2026-03-14 09:44:27.203707 | controller |       \         exit 1\n        fi\n        if [ \"$notif_vhost\" != \"watcher-notifications\"
2026-03-14 09:44:27.203714 | controller |       ]; then\n          echo \"Expected notification-transport vhost 'watcher-notifications',
2026-03-14 09:44:27.203721 | controller |       found '$notif_vhost'\"\n          exit 1\n        fi\n        echo \"Notification
2026-03-14 09:44:27.203727 | controller |       transport has correct user (watcher-notifications) and vhost (watcher-notifications)\"\n
2026-03-14 09:44:27.203737 | controller |       \       \n        # Verify that watcher.conf contains the notifications transport_url\n
2026-03-14 09:44:27.203744 | controller |       \       WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l \"service=watcher-api\"
2026-03-14 09:44:27.203750 | controller |       -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)\n        if
2026-03-14 09:44:27.203759 | controller |       [ -z \"${WATCHER_API_POD}\" ]; then\n          echo \"No watcher-api pod found\"\n
2026-03-14 09:44:27.203766 | controller |       \         exit 1\n        fi\n        # Verify RPC transport_url in DEFAULT section\n
2026-03-14 09:44:27.203773 | controller |       \       rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api
2026-03-14 09:44:27.203779 | controller |       -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E '^\\[DEFAULT\\]' -A
2026-03-14 09:44:27.203786 | controller |       50 | grep 'transport_url' | head -1 || true)\n        if [ -z \"$rpc_transport_url\"
2026-03-14 09:44:27.203792 | controller |       ]; then\n          echo \"transport_url not found in DEFAULT section\"\n          exit
2026-03-14 09:44:27.203799 | controller |       1\n        fi\n        echo \"Found RPC transport_url: $rpc_transport_url\"\n        \n
2026-03-14 09:44:27.203805 | controller |       \       # Verify the RPC transport_url contains the correct vhost (watcher-rpc)\n
2026-03-14 09:44:27.203812 | controller |       \       if ! echo \"$rpc_transport_url\" | grep -q '/watcher-rpc'; then\n          echo
2026-03-14 09:44:27.203818 | controller |       \"RPC transport_url does not contain expected vhost '/watcher-rpc'\"\n          exit
2026-03-14 09:44:27.203825 | controller |       1\n        fi\n        echo \"Successfully verified vhost 'watcher-rpc' in RPC transport_url\"\n
2026-03-14 09:44:27.203831 | controller |       \       \n        # Verify the RPC transport_url contains the correct username (watcher-rpc)\n
2026-03-14 09:44:27.203838 | controller |       \       if ! echo \"$rpc_transport_url\" | grep -q 'watcher-rpc:'; then\n          echo
2026-03-14 09:44:27.203845 | controller |       \"RPC transport_url does not contain expected username 'watcher-rpc:'\"\n          exit
2026-03-14 09:44:27.203851 | controller |       1\n        fi\n        echo \"Successfully verified username 'watcher-rpc' in RPC
2026-03-14 09:44:27.203858 | controller |       transport_url\"\n        \n        # Verify oslo_messaging_notifications section
2026-03-14 09:44:27.203864 | controller |       has transport_url configured\n        notif_transport_url=$(kubectl exec -n $NAMESPACE
2026-03-14 09:44:27.203871 | controller |       ${WATCHER_API_POD} -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf
2026-03-14 09:44:27.203877 | controller |       | grep -A 5 '\\[oslo_messaging_notifications\\]' | grep 'transport_url' || true)\n
2026-03-14 09:44:27.203884 | controller |       \       if [ -z \"$notif_transport_url\" ]; then\n          echo \"transport_url
2026-03-14 09:44:27.203890 | controller |       not found in oslo_messaging_notifications section\"\n          exit 1\n        fi\n
2026-03-14 09:44:27.203897 | controller |       \       echo \"Found notifications transport_url: $notif_transport_url\"\n        \n
2026-03-14 09:44:27.203904 | controller |       \       # Verify the notifications transport_url contains the correct vhost (watcher-notifications)\n
2026-03-14 09:44:27.203911 | controller |       \       if ! echo \"$notif_transport_url\" | grep -q '/watcher-notifications'; then\n
2026-03-14 09:44:27.203919 | controller |       \         echo \"Notifications transport_url does not contain expected vhost '/watcher-notifications'\"\n
2026-03-14 09:44:27.203926 | controller |       \         exit 1\n        fi\n        echo \"Successfully verified vhost 'watcher-notifications'
2026-03-14 09:44:27.203934 | controller |       in notifications transport_url\"\n        \n        # Verify the notifications transport_url
2026-03-14 09:44:27.203941 | controller |       contains the correct username (watcher-notifications)\n        if ! echo \"$notif_transport_url\"
2026-03-14 09:44:27.203947 | controller |       | grep -q 'watcher-notifications:'; then\n          echo \"Notifications transport_url
2026-03-14 09:44:27.203954 | controller |       does not contain expected username 'watcher-notifications:'\"\n          exit 1\n
2026-03-14 09:44:27.203960 | controller |       \       fi\n        echo \"Successfully verified username 'watcher-notifications'
2026-03-14 09:44:27.203967 | controller |       in notifications transport_url\"\n        \n        exit 0\n        ]\n    logger.go:42:
2026-03-14 09:44:27.203973 | controller |       09:39:14 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready watcher/watcher-kuttl
2026-03-14 09:44:27.203979 | controller |       -n watcher-kuttl-default --timeout=300s\n    logger.go:42: 09:39:39 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.203989 | controller |       | watcher.watcher.openstack.org/watcher-kuttl condition met\n    logger.go:42: 09:39:39
2026-03-14 09:44:27.203996 | controller |       | watcher-rmquser/1-deploy | + kubectl get watcher watcher-kuttl -n watcher-kuttl-default
2026-03-14 09:44:27.204011 | controller |       -o 'jsonpath={.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}'\n
2026-03-14 09:44:27.204018 | controller |       \   logger.go:42: 09:39:39 | watcher-rmquser/1-deploy | + grep -q True\n    logger.go:42:
2026-03-14 09:44:27.204024 | controller |       09:39:39 | watcher-rmquser/1-deploy | + echo 'WatcherNotificationTransportURLReady
2026-03-14 09:44:27.204031 | controller |       condition is True'\n    logger.go:42: 09:39:39 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady
2026-03-14 09:44:27.204037 | controller |       condition is True\n    logger.go:42: 09:39:39 | watcher-rmquser/1-deploy | ++ kubectl
2026-03-14 09:44:27.204046 | controller |       get transporturl -n watcher-kuttl-default -o name\n    logger.go:42: 09:39:39 |
2026-03-14 09:44:27.204053 | controller |       watcher-rmquser/1-deploy | ++ wc -l\n    logger.go:42: 09:39:39 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204059 | controller |       | ++ grep watcher-kuttl-watcher-transport\n    logger.go:42: 09:39:39 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204065 | controller |       | + transport_count=1\n    logger.go:42: 09:39:39 | watcher-rmquser/1-deploy | ++
2026-03-14 09:44:27.204071 | controller |       kubectl get transporturl -n watcher-kuttl-default -o name\n    logger.go:42: 09:39:39
2026-03-14 09:44:27.204077 | controller |       | watcher-rmquser/1-deploy | ++ wc -l\n    logger.go:42: 09:39:39 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204083 | controller |       | ++ grep watcher-kuttl-watcher-notification\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204089 | controller |       | + notification_transport_count=1\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204095 | controller |       | + '[' 1 -ne 1 ']'\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + '['
2026-03-14 09:44:27.204101 | controller |       1 -ne 1 ']'\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + echo 'Correctly
2026-03-14 09:44:27.204107 | controller |       found 2 TransportURLs (separate clusters: transport and notification)'\n    logger.go:42:
2026-03-14 09:44:27.204113 | controller |       09:39:40 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs (separate
2026-03-14 09:44:27.204119 | controller |       clusters: transport and notification)\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204125 | controller |       | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default
2026-03-14 09:44:27.204131 | controller |       -o 'jsonpath={.spec.username}'\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204137 | controller |       | + transport_user=watcher-rpc\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204143 | controller |       | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default
2026-03-14 09:44:27.204149 | controller |       -o 'jsonpath={.spec.vhost}'\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204155 | controller |       | + transport_vhost=watcher-rpc\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204161 | controller |       | + '[' watcher-rpc '!=' watcher-rpc ']'\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204167 | controller |       | + '[' watcher-rpc '!=' watcher-rpc ']'\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204173 | controller |       | + echo 'Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)'\n
2026-03-14 09:44:27.204180 | controller |       \   logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | Watcher transport has correct
2026-03-14 09:44:27.204186 | controller |       user (watcher-rpc) and vhost (watcher-rpc)\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204192 | controller |       | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications
2026-03-14 09:44:27.204198 | controller |       -n watcher-kuttl-default -o 'jsonpath={.spec.username}'\n    logger.go:42: 09:39:40
2026-03-14 09:44:27.204207 | controller |       | watcher-rmquser/1-deploy | + notif_user=watcher-notifications\n    logger.go:42:
2026-03-14 09:44:27.204213 | controller |       09:39:40 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications
2026-03-14 09:44:27.204220 | controller |       -n watcher-kuttl-default -o 'jsonpath={.spec.vhost}'\n    logger.go:42: 09:39:40
2026-03-14 09:44:27.204226 | controller |       | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications\n    logger.go:42:
2026-03-14 09:44:27.204232 | controller |       09:39:40 | watcher-rmquser/1-deploy | + '[' watcher-notifications '!=' watcher-notifications
2026-03-14 09:44:27.204238 | controller |       ']'\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + '[' watcher-notifications
2026-03-14 09:44:27.204245 | controller |       '!=' watcher-notifications ']'\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204251 | controller |       | + echo 'Notification transport has correct user (watcher-notifications) and vhost
2026-03-14 09:44:27.204257 | controller |       (watcher-notifications)'\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204262 | controller |       | Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)\n
2026-03-14 09:44:27.204269 | controller |       \   logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ kubectl get pods -n watcher-kuttl-default
2026-03-14 09:44:27.204275 | controller |       -l service=watcher-api -o custom-columns=:metadata.name --no-headers\n    logger.go:42:
2026-03-14 09:44:27.204304 | controller |       09:39:40 | watcher-rmquser/1-deploy | ++ head -1\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204316 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + WATCHER_API_POD=watcher-kuttl-api-0\n
2026-03-14 09:44:27.204322 | controller |       \   logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + '[' -z watcher-kuttl-api-0
2026-03-14 09:44:27.204328 | controller |       ']'\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ kubectl exec -n
2026-03-14 09:44:27.204335 | controller |       watcher-kuttl-default watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.204341 | controller |       \   logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ grep transport_url\n
2026-03-14 09:44:27.204347 | controller |       \   logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ grep -E '^\\[DEFAULT\\]'
2026-03-14 09:44:27.204353 | controller |       -A 50\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ head -1\n    logger.go:42:
2026-03-14 09:44:27.204359 | controller |       09:39:40 | watcher-rmquser/1-deploy | + rpc_transport_url='transport_url = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n
2026-03-14 09:44:27.204366 | controller |       \   logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + '[' -z 'transport_url
2026-03-14 09:44:27.204372 | controller |       = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'
2026-03-14 09:44:27.204381 | controller |       ']'\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + echo 'Found RPC transport_url:
2026-03-14 09:44:27.204388 | controller |       transport_url = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n
2026-03-14 09:44:27.204394 | controller |       \   logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | Found RPC transport_url:
2026-03-14 09:44:27.204401 | controller |       transport_url = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1\n
2026-03-14 09:44:27.204407 | controller |       \   logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + echo 'transport_url =
2026-03-14 09:44:27.204413 | controller |       rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n
2026-03-14 09:44:27.204422 | controller |       \   logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc\n
2026-03-14 09:44:27.204431 | controller |       \   logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + echo 'Successfully verified
2026-03-14 09:44:27.204437 | controller |       vhost '\\''watcher-rpc'\\'' in RPC transport_url'\n    logger.go:42: 09:39:40 |
2026-03-14 09:44:27.204443 | controller |       watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-rpc' in RPC transport_url\n
2026-03-14 09:44:27.204450 | controller |       \   logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + echo 'transport_url =
2026-03-14 09:44:27.204456 | controller |       rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n
2026-03-14 09:44:27.204462 | controller |       \   logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:\n
2026-03-14 09:44:27.204469 | controller |       \   logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + echo 'Successfully verified
2026-03-14 09:44:27.204475 | controller |       username '\\''watcher-rpc'\\'' in RPC transport_url'\n    logger.go:42: 09:39:40
2026-03-14 09:44:27.204481 | controller |       | watcher-rmquser/1-deploy | Successfully verified username 'watcher-rpc' in RPC
2026-03-14 09:44:27.204487 | controller |       transport_url\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ grep -A
2026-03-14 09:44:27.204493 | controller |       5 '\\[oslo_messaging_notifications\\]'\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204499 | controller |       | ++ grep transport_url\n    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy |
2026-03-14 09:44:27.204505 | controller |       ++ kubectl exec -n watcher-kuttl-default watcher-kuttl-api-0 -c watcher-api -- cat
2026-03-14 09:44:27.204512 | controller |       /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42: 09:39:41 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204518 | controller |       | + notif_transport_url='transport_url = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n
2026-03-14 09:44:27.204525 | controller |       \   logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | + '[' -z 'transport_url
2026-03-14 09:44:27.204531 | controller |       = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'
2026-03-14 09:44:27.204537 | controller |       ']'\n    logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | + echo 'Found notifications
2026-03-14 09:44:27.204543 | controller |       transport_url: transport_url = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n
2026-03-14 09:44:27.204550 | controller |       \   logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | Found notifications transport_url:
2026-03-14 09:44:27.204582 | controller |       transport_url = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1\n
2026-03-14 09:44:27.204591 | controller |       \   logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | + echo 'transport_url =
2026-03-14 09:44:27.204597 | controller |       rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n
2026-03-14 09:44:27.204604 | controller |       \   logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications\n
2026-03-14 09:44:27.204610 | controller |       \   logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | + echo 'Successfully verified
2026-03-14 09:44:27.204616 | controller |       vhost '\\''watcher-notifications'\\'' in notifications transport_url'\n    logger.go:42:
2026-03-14 09:44:27.204622 | controller |       09:39:41 | watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-notifications'
2026-03-14 09:44:27.204628 | controller |       in notifications transport_url\n    logger.go:42: 09:39:41 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204634 | controller |       | + grep -q watcher-notifications:\n    logger.go:42: 09:39:41 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204640 | controller |       | + echo 'transport_url = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n
2026-03-14 09:44:27.204650 | controller |       \   logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | + echo 'Successfully verified
2026-03-14 09:44:27.204656 | controller |       username '\\''watcher-notifications'\\'' in notifications transport_url'\n    logger.go:42:
2026-03-14 09:44:27.204661 | controller |       09:39:41 | watcher-rmquser/1-deploy | Successfully verified username 'watcher-notifications'
2026-03-14 09:44:27.204666 | controller |       in notifications transport_url\n    logger.go:42: 09:39:41 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.204672 | controller |       | + exit 0\n    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | running command:
2026-03-14 09:44:27.204679 | controller |       [sh -c set -euxo pipefail\n        \n        # Wait for Watcher to be Ready\n        kubectl
2026-03-14 09:44:27.204685 | controller |       wait --for=condition=Ready watcher/watcher-kuttl -n $NAMESPACE --timeout=300s\n
2026-03-14 09:44:27.204691 | controller |       \       \n        # Verify WatcherNotificationTransportURLReady condition exists
2026-03-14 09:44:27.204698 | controller |       and is True\n        kubectl get watcher watcher-kuttl -n $NAMESPACE -o jsonpath='{.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}'
2026-03-14 09:44:27.204704 | controller |       | grep -q \"True\"\n        echo \"WatcherNotificationTransportURLReady condition
2026-03-14 09:44:27.204710 | controller |       is True\"\n        \n        # Count TransportURL CRs - should be exactly 2 (one
2026-03-14 09:44:27.204717 | controller |       for messaging, one for notifications)\n        transport_count=$(kubectl get transporturl
2026-03-14 09:44:27.204723 | controller |       -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-transport\" | wc -l)\n        notification_transport_count=$(kubectl
2026-03-14 09:44:27.204729 | controller |       get transporturl -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-notification\"
2026-03-14 09:44:27.204738 | controller |       | wc -l)\n        \n        if [ \"$transport_count\" -ne \"1\" ]; then\n          echo
2026-03-14 09:44:27.204745 | controller |       \"Expected 1 watcher-transport TransportURL, found $transport_count\"\n          exit
2026-03-14 09:44:27.204752 | controller |       1\n        fi\n        \n        if [ \"$notification_transport_count\" -ne \"1\"
2026-03-14 09:44:27.204758 | controller |       ]; then\n          echo \"Expected 1 notification-transport TransportURL, found
2026-03-14 09:44:27.204764 | controller |       $notification_transport_count\"\n          exit 1\n        fi\n        \n        echo
2026-03-14 09:44:27.204770 | controller |       \"Correctly found 2 TransportURLs (separate clusters: transport and notification)\"\n
2026-03-14 09:44:27.204776 | controller |       \       \n        # Verify watcher-transport has correct user and vhost\n        transport_user=$(kubectl
2026-03-14 09:44:27.204783 | controller |       get transporturl watcher-kuttl-watcher-transport -n $NAMESPACE -o jsonpath='{.spec.username}')\n
2026-03-14 09:44:27.204791 | controller |       \       transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport
2026-03-14 09:44:27.204798 | controller |       -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n        if [ \"$transport_user\" !=
2026-03-14 09:44:27.204804 | controller |       \"watcher-rpc\" ]; then\n          echo \"Expected watcher-transport username 'watcher-rpc',
2026-03-14 09:44:27.204810 | controller |       found '$transport_user'\"\n          exit 1\n        fi\n        if [ \"$transport_vhost\"
2026-03-14 09:44:27.204817 | controller |       != \"watcher-rpc\" ]; then\n          echo \"Expected watcher-transport vhost 'watcher-rpc',
2026-03-14 09:44:27.204823 | controller |       found '$transport_vhost'\"\n          exit 1\n        fi\n        echo \"Watcher
2026-03-14 09:44:27.204829 | controller |       transport has correct user (watcher-rpc) and vhost (watcher-rpc)\"\n        \n        #
2026-03-14 09:44:27.204836 | controller |       Verify notification-transport has correct user and vhost\n        notif_user=$(kubectl
2026-03-14 09:44:27.204842 | controller |       get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE
2026-03-14 09:44:27.204857 | controller |       -o jsonpath='{.spec.username}')\n        notif_vhost=$(kubectl get transporturl
2026-03-14 09:44:27.204865 | controller |       watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n
2026-03-14 09:44:27.204879 | controller |       \       if [ \"$notif_user\" != \"watcher-notifications\" ]; then\n          echo
2026-03-14 09:44:27.204885 | controller |       \"Expected notification-transport username 'watcher-notifications', found '$notif_user'\"\n
2026-03-14 09:44:27.204891 | controller |       \         exit 1\n        fi\n        if [ \"$notif_vhost\" != \"watcher-notifications\"
2026-03-14 09:44:27.204898 | controller |       ]; then\n          echo \"Expected notification-transport vhost 'watcher-notifications',
2026-03-14 09:44:27.204904 | controller |       found '$notif_vhost'\"\n          exit 1\n        fi\n        echo \"Notification
2026-03-14 09:44:27.204910 | controller |       transport has correct user (watcher-notifications) and vhost (watcher-notifications)\"\n
2026-03-14 09:44:27.204916 | controller |       \       \n        # Verify that watcher.conf contains the notifications transport_url\n
2026-03-14 09:44:27.204922 | controller |       \       WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l \"service=watcher-api\"
2026-03-14 09:44:27.204928 | controller |       -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)\n        if
2026-03-14 09:44:27.204934 | controller |       [ -z \"${WATCHER_API_POD}\" ]; then\n          echo \"No watcher-api pod found\"\n
2026-03-14 09:44:27.204940 | controller |       \         exit 1\n        fi\n        # Verify RPC transport_url in DEFAULT section\n
2026-03-14 09:44:27.204945 | controller |       \       rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api
2026-03-14 09:44:27.204951 | controller |       -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E '^\\[DEFAULT\\]' -A
2026-03-14 09:44:27.204957 | controller |       50 | grep 'transport_url' | head -1 || true)\n        if [ -z \"$rpc_transport_url\"
2026-03-14 09:44:27.204963 | controller |       ]; then\n          echo \"transport_url not found in DEFAULT section\"\n          exit
2026-03-14 09:44:27.204969 | controller |       1\n        fi\n        echo \"Found RPC transport_url: $rpc_transport_url\"\n        \n
2026-03-14 09:44:27.204975 | controller |       \       # Verify the RPC transport_url contains the correct vhost (watcher-rpc)\n
2026-03-14 09:44:27.204982 | controller |       \       if ! echo \"$rpc_transport_url\" | grep -q '/watcher-rpc'; then\n          echo
2026-03-14 09:44:27.204988 | controller |       \"RPC transport_url does not contain expected vhost '/watcher-rpc'\"\n          exit
2026-03-14 09:44:27.204995 | controller |       1\n        fi\n        echo \"Successfully verified vhost 'watcher-rpc' in RPC transport_url\"\n
2026-03-14 09:44:27.205000 | controller |       \       \n        # Verify the RPC transport_url contains the correct username (watcher-rpc)\n
2026-03-14 09:44:27.205006 | controller |       \       if ! echo \"$rpc_transport_url\" | grep -q 'watcher-rpc:'; then\n          echo
2026-03-14 09:44:27.205012 | controller |       \"RPC transport_url does not contain expected username 'watcher-rpc:'\"\n          exit
2026-03-14 09:44:27.205018 | controller |       1\n        fi\n        echo \"Successfully verified username 'watcher-rpc' in RPC
2026-03-14 09:44:27.205024 | controller |       transport_url\"\n        \n        # Verify oslo_messaging_notifications section
2026-03-14 09:44:27.205030 | controller |       has transport_url configured\n        notif_transport_url=$(kubectl exec -n $NAMESPACE
2026-03-14 09:44:27.205036 | controller |       ${WATCHER_API_POD} -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf
2026-03-14 09:44:27.205046 | controller |       | grep -A 5 '\\[oslo_messaging_notifications\\]' | grep 'transport_url' || true)\n
2026-03-14 09:44:27.205052 | controller |       \       if [ -z \"$notif_transport_url\" ]; then\n          echo \"transport_url
2026-03-14 09:44:27.205058 | controller |       not found in oslo_messaging_notifications section\"\n          exit 1\n        fi\n
2026-03-14 09:44:27.205064 | controller |       \       echo \"Found notifications transport_url: $notif_transport_url\"\n        \n
2026-03-14 09:44:27.205070 | controller |       \       # Verify the notifications transport_url contains the correct vhost (watcher-notifications)\n
2026-03-14 09:44:27.205076 | controller |       \       if ! echo \"$notif_transport_url\" | grep -q '/watcher-notifications'; then\n
2026-03-14 09:44:27.205085 | controller |       \         echo \"Notifications transport_url does not contain expected vhost '/watcher-notifications'\"\n
2026-03-14 09:44:27.205091 | controller |       \         exit 1\n        fi\n        echo \"Successfully verified vhost 'watcher-notifications'
2026-03-14 09:44:27.205098 | controller |       in notifications transport_url\"\n        \n        # Verify the notifications transport_url
2026-03-14 09:44:27.205104 | controller |       contains the correct username (watcher-notifications)\n        if ! echo \"$notif_transport_url\"
2026-03-14 09:44:27.205110 | controller |       | grep -q 'watcher-notifications:'; then\n          echo \"Notifications transport_url
2026-03-14 09:44:27.205116 | controller |       does not contain expected username 'watcher-notifications:'\"\n          exit 1\n
2026-03-14 09:44:27.205122 | controller |       \       fi\n        echo \"Successfully verified username 'watcher-notifications'
2026-03-14 09:44:27.205129 | controller |       in notifications transport_url\"\n        \n        exit 0\n        ]\n    logger.go:42:
2026-03-14 09:44:27.205144 | controller |       09:39:42 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready watcher/watcher-kuttl
2026-03-14 09:44:27.205151 | controller |       -n watcher-kuttl-default --timeout=300s\n    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205157 | controller |       | watcher.watcher.openstack.org/watcher-kuttl condition met\n    logger.go:42: 09:39:42
2026-03-14 09:44:27.205164 | controller |       | watcher-rmquser/1-deploy | + kubectl get watcher watcher-kuttl -n watcher-kuttl-default
2026-03-14 09:44:27.205170 | controller |       -o 'jsonpath={.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}'\n
2026-03-14 09:44:27.205179 | controller |       \   logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | + grep -q True\n    logger.go:42:
2026-03-14 09:44:27.205186 | controller |       09:39:42 | watcher-rmquser/1-deploy | + echo 'WatcherNotificationTransportURLReady
2026-03-14 09:44:27.205192 | controller |       condition is True'\n    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady
2026-03-14 09:44:27.205199 | controller |       condition is True\n    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | ++ kubectl
2026-03-14 09:44:27.205205 | controller |       get transporturl -n watcher-kuttl-default -o name\n    logger.go:42: 09:39:42 |
2026-03-14 09:44:27.205212 | controller |       watcher-rmquser/1-deploy | ++ wc -l\n    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205219 | controller |       | ++ grep watcher-kuttl-watcher-transport\n    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205225 | controller |       | + transport_count=1\n    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | ++
2026-03-14 09:44:27.205231 | controller |       kubectl get transporturl -n watcher-kuttl-default -o name\n    logger.go:42: 09:39:42
2026-03-14 09:44:27.205238 | controller |       | watcher-rmquser/1-deploy | ++ wc -l\n    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205244 | controller |       | ++ grep watcher-kuttl-watcher-notification\n    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205251 | controller |       | + notification_transport_count=1\n    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205257 | controller |       | + '[' 1 -ne 1 ']'\n    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | + '['
2026-03-14 09:44:27.205264 | controller |       1 -ne 1 ']'\n    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | + echo 'Correctly
2026-03-14 09:44:27.205271 | controller |       found 2 TransportURLs (separate clusters: transport and notification)'\n    logger.go:42:
2026-03-14 09:44:27.205277 | controller |       09:39:42 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs (separate
2026-03-14 09:44:27.205284 | controller |       clusters: transport and notification)\n    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205310 | controller |       | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default
2026-03-14 09:44:27.205318 | controller |       -o 'jsonpath={.spec.username}'\n    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205325 | controller |       | + transport_user=watcher-rpc\n    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205334 | controller |       | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default
2026-03-14 09:44:27.205341 | controller |       -o 'jsonpath={.spec.vhost}'\n    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205348 | controller |       | + transport_vhost=watcher-rpc\n    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205354 | controller |       | + '[' watcher-rpc '!=' watcher-rpc ']'\n    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205361 | controller |       | + '[' watcher-rpc '!=' watcher-rpc ']'\n    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205367 | controller |       | + echo 'Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)'\n
2026-03-14 09:44:27.205374 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | Watcher transport has correct
2026-03-14 09:44:27.205380 | controller |       user (watcher-rpc) and vhost (watcher-rpc)\n    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205387 | controller |       | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications
2026-03-14 09:44:27.205393 | controller |       -n watcher-kuttl-default -o 'jsonpath={.spec.username}'\n    logger.go:42: 09:39:43
2026-03-14 09:44:27.205399 | controller |       | watcher-rmquser/1-deploy | + notif_user=watcher-notifications\n    logger.go:42:
2026-03-14 09:44:27.205406 | controller |       09:39:43 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications
2026-03-14 09:44:27.205412 | controller |       -n watcher-kuttl-default -o 'jsonpath={.spec.vhost}'\n    logger.go:42: 09:39:43
2026-03-14 09:44:27.205418 | controller |       | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications\n    logger.go:42:
2026-03-14 09:44:27.205424 | controller |       09:39:43 | watcher-rmquser/1-deploy | + '[' watcher-notifications '!=' watcher-notifications
2026-03-14 09:44:27.205433 | controller |       ']'\n    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + '[' watcher-notifications
2026-03-14 09:44:27.205440 | controller |       '!=' watcher-notifications ']'\n    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205447 | controller |       | + echo 'Notification transport has correct user (watcher-notifications) and vhost
2026-03-14 09:44:27.205462 | controller |       (watcher-notifications)'\n    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205469 | controller |       | Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)\n
2026-03-14 09:44:27.205475 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | ++ kubectl get pods -n watcher-kuttl-default
2026-03-14 09:44:27.205482 | controller |       -l service=watcher-api -o custom-columns=:metadata.name --no-headers\n    logger.go:42:
2026-03-14 09:44:27.205488 | controller |       09:39:43 | watcher-rmquser/1-deploy | ++ head -1\n    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205494 | controller |       | ++ grep -v '^$'\n    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + WATCHER_API_POD=watcher-kuttl-api-0\n
2026-03-14 09:44:27.205501 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + '[' -z watcher-kuttl-api-0
2026-03-14 09:44:27.205507 | controller |       ']'\n    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | ++ grep transport_url\n
2026-03-14 09:44:27.205514 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | ++ grep -E '^\\[DEFAULT\\]'
2026-03-14 09:44:27.205520 | controller |       -A 50\n    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | ++ head -1\n    logger.go:42:
2026-03-14 09:44:27.205526 | controller |       09:39:43 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default watcher-kuttl-api-0
2026-03-14 09:44:27.205532 | controller |       -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf\n    logger.go:42:
2026-03-14 09:44:27.205539 | controller |       09:39:43 | watcher-rmquser/1-deploy | + rpc_transport_url='transport_url = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n
2026-03-14 09:44:27.205545 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + '[' -z 'transport_url
2026-03-14 09:44:27.205555 | controller |       = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'
2026-03-14 09:44:27.205561 | controller |       ']'\n    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo 'Found RPC transport_url:
2026-03-14 09:44:27.205568 | controller |       transport_url = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n
2026-03-14 09:44:27.205574 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | Found RPC transport_url:
2026-03-14 09:44:27.205580 | controller |       transport_url = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1\n
2026-03-14 09:44:27.205587 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo 'transport_url =
2026-03-14 09:44:27.205647 | controller |       rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n
2026-03-14 09:44:27.205655 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc\n
2026-03-14 09:44:27.205662 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo 'Successfully verified
2026-03-14 09:44:27.205668 | controller |       vhost '\\''watcher-rpc'\\'' in RPC transport_url'\n    logger.go:42: 09:39:43 |
2026-03-14 09:44:27.205675 | controller |       watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-rpc' in RPC transport_url\n
2026-03-14 09:44:27.205681 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo 'transport_url =
2026-03-14 09:44:27.205688 | controller |       rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n
2026-03-14 09:44:27.205714 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:\n
2026-03-14 09:44:27.205753 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo 'Successfully verified
2026-03-14 09:44:27.205761 | controller |       username '\\''watcher-rpc'\\'' in RPC transport_url'\n    logger.go:42: 09:39:43
2026-03-14 09:44:27.205768 | controller |       | watcher-rmquser/1-deploy | Successfully verified username 'watcher-rpc' in RPC
2026-03-14 09:44:27.205775 | controller |       transport_url\n    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | ++ kubectl
2026-03-14 09:44:27.205781 | controller |       exec -n watcher-kuttl-default watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.205788 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | ++ grep transport_url\n
2026-03-14 09:44:27.205794 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | ++ grep -A 5 '\\[oslo_messaging_notifications\\]'\n
2026-03-14 09:44:27.205801 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + notif_transport_url='transport_url
2026-03-14 09:44:27.205807 | controller |       = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n
2026-03-14 09:44:27.205814 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + '[' -z 'transport_url
2026-03-14 09:44:27.205820 | controller |       = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'
2026-03-14 09:44:27.205836 | controller |       ']'\n    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo 'Found notifications
2026-03-14 09:44:27.205844 | controller |       transport_url: transport_url = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n
2026-03-14 09:44:27.205850 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | Found notifications transport_url:
2026-03-14 09:44:27.205857 | controller |       transport_url = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1\n
2026-03-14 09:44:27.205867 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo 'transport_url =
2026-03-14 09:44:27.205873 | controller |       rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n
2026-03-14 09:44:27.205880 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications\n
2026-03-14 09:44:27.205886 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo 'Successfully verified
2026-03-14 09:44:27.205892 | controller |       vhost '\\''watcher-notifications'\\'' in notifications transport_url'\n    logger.go:42:
2026-03-14 09:44:27.205898 | controller |       09:39:43 | watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-notifications'
2026-03-14 09:44:27.205905 | controller |       in notifications transport_url\n    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205911 | controller |       | + grep -q watcher-notifications:\n    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205917 | controller |       | + echo 'transport_url = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n
2026-03-14 09:44:27.205924 | controller |       \   logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo 'Successfully verified
2026-03-14 09:44:27.205930 | controller |       username '\\''watcher-notifications'\\'' in notifications transport_url'\n    logger.go:42:
2026-03-14 09:44:27.205936 | controller |       09:39:43 | watcher-rmquser/1-deploy | Successfully verified username 'watcher-notifications'
2026-03-14 09:44:27.205943 | controller |       in notifications transport_url\n    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy
2026-03-14 09:44:27.205949 | controller |       | + exit 0\n    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | test step completed
2026-03-14 09:44:27.205959 | controller |       1-deploy\n    logger.go:42: 09:39:43 | watcher-rmquser/2-cleanup-watcher | starting
2026-03-14 09:44:27.205966 | controller |       test step 2-cleanup-watcher\n    logger.go:42: 09:39:49 | watcher-rmquser/2-cleanup-watcher
2026-03-14 09:44:27.205972 | controller |       | test step completed 2-cleanup-watcher\n    logger.go:42: 09:39:49 | watcher-rmquser
2026-03-14 09:44:27.205979 | controller |       | skipping kubernetes event logging\n=== CONT  kuttl/harness/watcher\n    logger.go:42:
2026-03-14 09:44:27.205985 | controller |       09:39:49 | watcher | Skipping creation of user-supplied namespace: watcher-kuttl-default\n
2026-03-14 09:44:27.205994 | controller |       \   logger.go:42: 09:39:49 | watcher/0-cleanup-watcher | starting test step 0-cleanup-watcher\n
2026-03-14 09:44:27.206001 | controller |       \   logger.go:42: 09:39:49 | watcher/0-cleanup-watcher | test step completed 0-cleanup-watcher\n
2026-03-14 09:44:27.206007 | controller |       \   logger.go:42: 09:39:49 | watcher/1-deploy-with-defaults | starting test step
2026-03-14 09:44:27.206014 | controller |       1-deploy-with-defaults\n    logger.go:42: 09:39:49 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206020 | controller |       | Watcher:watcher-kuttl-default/watcher-kuttl created\n    logger.go:42: 09:39:49
2026-03-14 09:44:27.206027 | controller |       | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:44:27.206034 | controller |       \       oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.206040 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc exec
2026-03-14 09:44:27.206046 | controller |       -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.206053 | controller |       -c Type -c ID | grep watcher| awk '{print $1}')\n        [ $(oc get -n watcher-kuttl-default
2026-03-14 09:44:27.206060 | controller |       keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [
2026-03-14 09:44:27.206066 | controller |       -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\"
2026-03-14 09:44:27.206073 | controller |       ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.206080 | controller |       -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n        [ \"$(oc
2026-03-14 09:44:27.206089 | controller |       get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64
2026-03-14 09:44:27.206095 | controller |       -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2
2026-03-14 09:44:27.206102 | controller |       ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.206117 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\"
2026-03-14 09:44:27.206125 | controller |       -ge 1 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:44:27.206131 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:44:27.206138 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:44:27.206144 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:44:27.206151 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:44:27.206157 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:44:27.206164 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:44:27.206171 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:44:27.206177 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:44:27.206184 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:39:49 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206190 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.206197 | controller |       value -c Name -c Type\n    logger.go:42: 09:39:49 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206204 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:39:52 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206210 | controller |       | + '[' 0 == 1 ']'\n    logger.go:42: 09:39:53 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206217 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.206223 | controller |       openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c
2026-03-14 09:44:27.206230 | controller |       ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.206236 | controller |       -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print
2026-03-14 09:44:27.206243 | controller |       $1}')\n        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.206249 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl
2026-03-14 09:44:27.206256 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        [ \"$(oc get -n watcher-kuttl-default
2026-03-14 09:44:27.206265 | controller |       secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep
2026-03-14 09:44:27.206272 | controller |       -c 'ssl=1')\" == 1 ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.206279 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\"
2026-03-14 09:44:27.206286 | controller |       == 2 ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.206312 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\"
2026-03-14 09:44:27.206319 | controller |       -ge 1 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:44:27.206326 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:44:27.206335 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:44:27.206342 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:44:27.206348 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:44:27.206355 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:44:27.206361 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:44:27.206368 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:44:27.206375 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:44:27.206381 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:39:53 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206388 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.206394 | controller |       value -c Name -c Type\n    logger.go:42: 09:39:53 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206401 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:39:55 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206408 | controller |       | + '[' 0 == 1 ']'\n    logger.go:42: 09:39:56 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206414 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.206421 | controller |       openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c
2026-03-14 09:44:27.206437 | controller |       ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.206445 | controller |       -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print
2026-03-14 09:44:27.206451 | controller |       $1}')\n        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.206458 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl
2026-03-14 09:44:27.206464 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        [ \"$(oc get -n watcher-kuttl-default
2026-03-14 09:44:27.206471 | controller |       secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep
2026-03-14 09:44:27.206477 | controller |       -c 'ssl=1')\" == 1 ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.206484 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\"
2026-03-14 09:44:27.206490 | controller |       == 2 ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.206497 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\"
2026-03-14 09:44:27.206503 | controller |       -ge 1 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:44:27.206509 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:44:27.206515 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:44:27.206522 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:44:27.206528 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:44:27.206534 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:44:27.206540 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:44:27.206546 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:44:27.206553 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:44:27.206562 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:39:56 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206569 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.206576 | controller |       value -c Name -c Type\n    logger.go:42: 09:39:56 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206583 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:39:58 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206589 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:39:58 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206595 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:39:58 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206601 | controller |       | ++ grep watcher\n    logger.go:42: 09:39:58 | watcher/1-deploy-with-defaults |
2026-03-14 09:44:27.206611 | controller |       ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.206617 | controller |       value -c Name -c Type -c ID\n    logger.go:42: 09:40:00 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206624 | controller |       | + SERVICEID=55dad13ac5da4c7fbfa109c6b4aa619f\n    logger.go:42: 09:40:00 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206630 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.206636 | controller |       \   logger.go:42: 09:40:00 | watcher/1-deploy-with-defaults | + '[' 55dad13ac5da4c7fbfa109c6b4aa619f
2026-03-14 09:44:27.206643 | controller |       == 55dad13ac5da4c7fbfa109c6b4aa619f ']'\n    logger.go:42: 09:40:00 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206649 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:44:27.206655 | controller |       \   logger.go:42: 09:40:00 | watcher/1-deploy-with-defaults | + '[' -n '' ']'\n
2026-03-14 09:44:27.206662 | controller |       \   logger.go:42: 09:40:01 | watcher/1-deploy-with-defaults | running command: [sh
2026-03-14 09:44:27.206669 | controller |       -c set -euxo pipefail\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.206675 | controller |       -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:44:27.206681 | controller |       \       SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:44:27.206688 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n
2026-03-14 09:44:27.206694 | controller |       \       [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.206700 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl
2026-03-14 09:44:27.206706 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        [ \"$(oc get -n watcher-kuttl-default
2026-03-14 09:44:27.206712 | controller |       secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep
2026-03-14 09:44:27.206733 | controller |       -c 'ssl=1')\" == 1 ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.206741 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\"
2026-03-14 09:44:27.206748 | controller |       == 2 ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.206754 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\"
2026-03-14 09:44:27.206760 | controller |       -ge 1 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:44:27.206767 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:44:27.206773 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:44:27.206779 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:44:27.206785 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:44:27.206794 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:44:27.206801 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:44:27.206807 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:44:27.206813 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:44:27.206820 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:40:01 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206826 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.206832 | controller |       value -c Name -c Type\n    logger.go:42: 09:40:01 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206838 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:40:03 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206845 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:40:03 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206851 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:40:03 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206857 | controller |       | ++ grep watcher\n    logger.go:42: 09:40:03 | watcher/1-deploy-with-defaults |
2026-03-14 09:44:27.206864 | controller |       ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.206870 | controller |       value -c Name -c Type -c ID\n    logger.go:42: 09:40:05 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206876 | controller |       | + SERVICEID=55dad13ac5da4c7fbfa109c6b4aa619f\n    logger.go:42: 09:40:05 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206882 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.206889 | controller |       \   logger.go:42: 09:40:05 | watcher/1-deploy-with-defaults | + '[' 55dad13ac5da4c7fbfa109c6b4aa619f
2026-03-14 09:44:27.206895 | controller |       == 55dad13ac5da4c7fbfa109c6b4aa619f ']'\n    logger.go:42: 09:40:05 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206901 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:44:27.206907 | controller |       \   logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.206915 | controller |       ']'\n    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | ++ base64 -d\n
2026-03-14 09:44:27.206926 | controller |       \   logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.206933 | controller |       secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n    logger.go:42:
2026-03-14 09:44:27.206941 | controller |       09:40:06 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1\n    logger.go:42:
2026-03-14 09:44:27.206947 | controller |       09:40:06 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:44:27.206954 | controller |       09:40:06 | watcher/1-deploy-with-defaults | ++ base64 -d\n    logger.go:42: 09:40:06
2026-03-14 09:44:27.206960 | controller |       | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.206967 | controller |       -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.206973 | controller |       | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n    logger.go:42:
2026-03-14 09:44:27.206980 | controller |       09:40:06 | watcher/1-deploy-with-defaults | + '[' 2 == 2 ']'\n    logger.go:42:
2026-03-14 09:44:27.206989 | controller |       09:40:06 | watcher/1-deploy-with-defaults | ++ grep -c 'region_name = regionOne'\n
2026-03-14 09:44:27.206996 | controller |       \   logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | ++ base64 -d\n    logger.go:42:
2026-03-14 09:44:27.207002 | controller |       09:40:06 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret
2026-03-14 09:44:27.207020 | controller |       watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42:
2026-03-14 09:44:27.207028 | controller |       09:40:06 | watcher/1-deploy-with-defaults | + '[' 9 -ge 1 ']'\n    logger.go:42:
2026-03-14 09:44:27.207034 | controller |       09:40:06 | watcher/1-deploy-with-defaults | ++ oc get pods -n openstack-operators
2026-03-14 09:44:27.207041 | controller |       -o name -l openstack.org/operator-name=watcher\n    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207048 | controller |       | + '[' '' == '' ']'\n    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207054 | controller |       | + exit 0\n    logger.go:42: 09:40:07 | watcher/1-deploy-with-defaults | running
2026-03-14 09:44:27.207061 | controller |       command: [sh -c set -euxo pipefail\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.207067 | controller |       -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:44:27.207073 | controller |       \       SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:44:27.207080 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n
2026-03-14 09:44:27.207086 | controller |       \       [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.207092 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl
2026-03-14 09:44:27.207099 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        [ \"$(oc get -n watcher-kuttl-default
2026-03-14 09:44:27.207105 | controller |       secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep
2026-03-14 09:44:27.207112 | controller |       -c 'ssl=1')\" == 1 ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.207118 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\"
2026-03-14 09:44:27.207125 | controller |       == 2 ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.207131 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\"
2026-03-14 09:44:27.207138 | controller |       -ge 1 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:44:27.207144 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:44:27.207150 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:44:27.207156 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:44:27.207163 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:44:27.207169 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:44:27.207176 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:44:27.207182 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:44:27.207188 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:44:27.207195 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:40:07 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207202 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.207208 | controller |       value -c Name -c Type\n    logger.go:42: 09:40:07 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207215 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:40:09 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207221 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:40:09 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207228 | controller |       | ++ grep watcher\n    logger.go:42: 09:40:09 | watcher/1-deploy-with-defaults |
2026-03-14 09:44:27.207238 | controller |       ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.207244 | controller |       value -c Name -c Type -c ID\n    logger.go:42: 09:40:09 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207251 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207258 | controller |       | + SERVICEID=55dad13ac5da4c7fbfa109c6b4aa619f\n    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207264 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.207271 | controller |       \   logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | + '[' 55dad13ac5da4c7fbfa109c6b4aa619f
2026-03-14 09:44:27.207278 | controller |       == 55dad13ac5da4c7fbfa109c6b4aa619f ']'\n    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207284 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:44:27.207313 | controller |       \   logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.207330 | controller |       ']'\n    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | ++ base64 -d\n
2026-03-14 09:44:27.207338 | controller |       \   logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.207344 | controller |       secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n    logger.go:42:
2026-03-14 09:44:27.207350 | controller |       09:40:11 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1\n    logger.go:42:
2026-03-14 09:44:27.207357 | controller |       09:40:11 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:44:27.207363 | controller |       09:40:11 | watcher/1-deploy-with-defaults | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n
2026-03-14 09:44:27.207373 | controller |       \   logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | ++ base64 -d\n    logger.go:42:
2026-03-14 09:44:27.207381 | controller |       09:40:11 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret
2026-03-14 09:44:27.207387 | controller |       watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42:
2026-03-14 09:44:27.207394 | controller |       09:40:11 | watcher/1-deploy-with-defaults | + '[' 2 == 2 ']'\n    logger.go:42:
2026-03-14 09:44:27.207400 | controller |       09:40:11 | watcher/1-deploy-with-defaults | ++ base64 -d\n    logger.go:42: 09:40:11
2026-03-14 09:44:27.207406 | controller |       | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.207412 | controller |       -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207418 | controller |       | ++ grep -c 'region_name = regionOne'\n    logger.go:42: 09:40:12 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207424 | controller |       | + '[' 9 -ge 1 ']'\n    logger.go:42: 09:40:12 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207430 | controller |       | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:44:27.207436 | controller |       \   logger.go:42: 09:40:12 | watcher/1-deploy-with-defaults | + '[' '' == '' ']'\n
2026-03-14 09:44:27.207442 | controller |       \   logger.go:42: 09:40:12 | watcher/1-deploy-with-defaults | + exit 0\n    logger.go:42:
2026-03-14 09:44:27.207447 | controller |       09:40:13 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:44:27.207453 | controller |       \       oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.207459 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc exec
2026-03-14 09:44:27.207465 | controller |       -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.207470 | controller |       -c Type -c ID | grep watcher| awk '{print $1}')\n        [ $(oc get -n watcher-kuttl-default
2026-03-14 09:44:27.207479 | controller |       keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [
2026-03-14 09:44:27.207485 | controller |       -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\"
2026-03-14 09:44:27.207491 | controller |       ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.207497 | controller |       -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n        [ \"$(oc
2026-03-14 09:44:27.207503 | controller |       get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64
2026-03-14 09:44:27.207509 | controller |       -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2
2026-03-14 09:44:27.207515 | controller |       ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.207522 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\"
2026-03-14 09:44:27.207528 | controller |       -ge 1 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:44:27.207534 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:44:27.207540 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:44:27.207546 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:44:27.207553 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:44:27.207559 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:44:27.207565 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:44:27.207572 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:44:27.207578 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:44:27.207584 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:40:13 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207590 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.207596 | controller |       value -c Name -c Type\n    logger.go:42: 09:40:13 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207611 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:40:15 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207618 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:40:15 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207625 | controller |       | ++ grep watcher\n    logger.go:42: 09:40:15 | watcher/1-deploy-with-defaults |
2026-03-14 09:44:27.207631 | controller |       ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.207637 | controller |       value -c Name -c Type -c ID\n    logger.go:42: 09:40:15 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207643 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207650 | controller |       | + SERVICEID=55dad13ac5da4c7fbfa109c6b4aa619f\n    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207656 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.207662 | controller |       \   logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | + '[' 55dad13ac5da4c7fbfa109c6b4aa619f
2026-03-14 09:44:27.207669 | controller |       == 55dad13ac5da4c7fbfa109c6b4aa619f ']'\n    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207675 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:44:27.207681 | controller |       \   logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.207693 | controller |       ']'\n    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | ++ base64 -d\n
2026-03-14 09:44:27.207699 | controller |       \   logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.207705 | controller |       secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n    logger.go:42:
2026-03-14 09:44:27.207710 | controller |       09:40:17 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1\n    logger.go:42:
2026-03-14 09:44:27.207716 | controller |       09:40:17 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:44:27.207722 | controller |       09:40:17 | watcher/1-deploy-with-defaults | ++ base64 -d\n    logger.go:42: 09:40:17
2026-03-14 09:44:27.207728 | controller |       | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.207734 | controller |       -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207739 | controller |       | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n    logger.go:42:
2026-03-14 09:44:27.207745 | controller |       09:40:17 | watcher/1-deploy-with-defaults | + '[' 2 == 2 ']'\n    logger.go:42:
2026-03-14 09:44:27.207751 | controller |       09:40:17 | watcher/1-deploy-with-defaults | ++ base64 -d\n    logger.go:42: 09:40:17
2026-03-14 09:44:27.207756 | controller |       | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.207761 | controller |       -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207766 | controller |       | ++ grep -c 'region_name = regionOne'\n    logger.go:42: 09:40:18 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207770 | controller |       | + '[' 9 -ge 1 ']'\n    logger.go:42: 09:40:18 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207774 | controller |       | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:44:27.207778 | controller |       \   logger.go:42: 09:40:18 | watcher/1-deploy-with-defaults | + '[' '' == '' ']'\n
2026-03-14 09:44:27.207782 | controller |       \   logger.go:42: 09:40:18 | watcher/1-deploy-with-defaults | + exit 0\n    logger.go:42:
2026-03-14 09:44:27.207787 | controller |       09:40:19 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:44:27.207791 | controller |       \       oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.207795 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc exec
2026-03-14 09:44:27.207799 | controller |       -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.207803 | controller |       -c Type -c ID | grep watcher| awk '{print $1}')\n        [ $(oc get -n watcher-kuttl-default
2026-03-14 09:44:27.207807 | controller |       keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [
2026-03-14 09:44:27.207812 | controller |       -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\"
2026-03-14 09:44:27.207816 | controller |       ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.207820 | controller |       -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n        [ \"$(oc
2026-03-14 09:44:27.207824 | controller |       get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64
2026-03-14 09:44:27.207828 | controller |       -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2
2026-03-14 09:44:27.207840 | controller |       ]\n        [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.207845 | controller |       -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\"
2026-03-14 09:44:27.207849 | controller |       -ge 1 ]\n        # If we are running the container locally, skip following test\n
2026-03-14 09:44:27.207854 | controller |       \       if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:44:27.207860 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:44:27.207864 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:44:27.207869 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:44:27.207873 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:44:27.207877 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:44:27.207881 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:44:27.207885 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:44:27.207890 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:40:19 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207894 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.207898 | controller |       value -c Name -c Type\n    logger.go:42: 09:40:19 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207902 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:40:21 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207907 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:40:21 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207911 | controller |       | ++ grep watcher\n    logger.go:42: 09:40:21 | watcher/1-deploy-with-defaults |
2026-03-14 09:44:27.207917 | controller |       ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.207922 | controller |       value -c Name -c Type -c ID\n    logger.go:42: 09:40:21 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207926 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207930 | controller |       | + SERVICEID=55dad13ac5da4c7fbfa109c6b4aa619f\n    logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207934 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.207939 | controller |       \   logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults | + '[' 55dad13ac5da4c7fbfa109c6b4aa619f
2026-03-14 09:44:27.207948 | controller |       == 55dad13ac5da4c7fbfa109c6b4aa619f ']'\n    logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.207953 | controller |       | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n
2026-03-14 09:44:27.207959 | controller |       \   logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.207964 | controller |       ']'\n    logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults | ++ base64 -d\n
2026-03-14 09:44:27.207969 | controller |       \   logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.207974 | controller |       secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n    logger.go:42:
2026-03-14 09:44:27.207979 | controller |       09:40:23 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1\n    logger.go:42:
2026-03-14 09:44:27.207984 | controller |       09:40:23 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:44:27.207988 | controller |       09:40:23 | watcher/1-deploy-with-defaults | ++ base64 -d\n    logger.go:42: 09:40:23
2026-03-14 09:44:27.207992 | controller |       | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.207997 | controller |       -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.208001 | controller |       | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n    logger.go:42:
2026-03-14 09:44:27.208005 | controller |       09:40:24 | watcher/1-deploy-with-defaults | + '[' 2 == 2 ']'\n    logger.go:42:
2026-03-14 09:44:27.208012 | controller |       09:40:24 | watcher/1-deploy-with-defaults | ++ grep -c 'region_name = regionOne'\n
2026-03-14 09:44:27.208016 | controller |       \   logger.go:42: 09:40:24 | watcher/1-deploy-with-defaults | ++ base64 -d\n    logger.go:42:
2026-03-14 09:44:27.208021 | controller |       09:40:24 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret
2026-03-14 09:44:27.208025 | controller |       watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n    logger.go:42:
2026-03-14 09:44:27.208029 | controller |       09:40:24 | watcher/1-deploy-with-defaults | + '[' 9 -ge 1 ']'\n    logger.go:42:
2026-03-14 09:44:27.208033 | controller |       09:40:24 | watcher/1-deploy-with-defaults | ++ oc get pods -n openstack-operators
2026-03-14 09:44:27.208043 | controller |       -o name -l openstack.org/operator-name=watcher\n    logger.go:42: 09:40:24 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.208048 | controller |       | + '[' '' == '' ']'\n    logger.go:42: 09:40:24 | watcher/1-deploy-with-defaults
2026-03-14 09:44:27.208053 | controller |       | + exit 0\n    logger.go:42: 09:40:24 | watcher/1-deploy-with-defaults | test step
2026-03-14 09:44:27.208057 | controller |       completed 1-deploy-with-defaults\n    logger.go:42: 09:40:24 | watcher/2-cleanup-watcher
2026-03-14 09:44:27.208061 | controller |       | starting test step 2-cleanup-watcher\n    logger.go:42: 09:40:24 | watcher/2-cleanup-watcher
2026-03-14 09:44:27.208065 | controller |       | test step completed 2-cleanup-watcher\n    logger.go:42: 09:40:24 | watcher/3-precreate-mariadbaccount
2026-03-14 09:44:27.208069 | controller |       | starting test step 3-precreate-mariadbaccount\n    logger.go:42: 09:40:24 | watcher/3-precreate-mariadbaccount
2026-03-14 09:44:27.208074 | controller |       | MariaDBAccount:watcher-kuttl-default/watcher-precreated created\n    logger.go:42:
2026-03-14 09:44:27.208078 | controller |       09:40:24 | watcher/3-precreate-mariadbaccount | test step completed 3-precreate-mariadbaccount\n
2026-03-14 09:44:27.208082 | controller |       \   logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account | starting
2026-03-14 09:44:27.208086 | controller |       test step 4-deploy-with-precreated-account\n    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208090 | controller |       | Secret:watcher-kuttl-default/custom-prometheus-config created\n    logger.go:42:
2026-03-14 09:44:27.208095 | controller |       09:40:24 | watcher/4-deploy-with-precreated-account | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:44:27.208099 | controller |       created\n    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.208103 | controller |       running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.208107 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.208111 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.208115 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.208120 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.208124 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.208128 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.208133 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.208137 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.208142 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.208146 | controller |       \   logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.208152 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208156 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.208160 | controller |       \   logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.208164 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.208168 | controller |       \   logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.208174 | controller |       -1\n    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.208178 | controller |       -v '^$'\n    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.208183 | controller |       + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208187 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208191 | controller |       | ++ grep -c '^# Global config'\n    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208197 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:44:27.208201 | controller |       \   logger.go:42: 09:40:25 | watcher/4-deploy-with-precreated-account | error: Internal
2026-03-14 09:44:27.208205 | controller |       error occurred: error executing command in container: container is not created or
2026-03-14 09:44:27.208209 | controller |       running\n    logger.go:42: 09:40:25 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.208213 | controller |       ++ echo\n    logger.go:42: 09:40:25 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.208218 | controller |       + '[' 0 == 1 ']'\n    logger.go:42: 09:40:29 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208228 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.208232 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.208236 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.208241 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.208245 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.208249 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.208253 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.208258 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.208262 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.208266 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.208270 | controller |       \   logger.go:42: 09:40:29 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.208275 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:29 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208279 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.208283 | controller |       \   logger.go:42: 09:40:29 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.208290 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.208312 | controller |       \   logger.go:42: 09:40:29 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.208317 | controller |       -1\n    logger.go:42: 09:40:29 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.208321 | controller |       -v '^$'\n    logger.go:42: 09:40:29 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.208325 | controller |       + APIPOD=\n    logger.go:42: 09:40:30 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208330 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.208334 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.208338 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.208342 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.208346 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.208350 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.208355 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.208359 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.208363 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.208367 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.208371 | controller |       \   logger.go:42: 09:40:30 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.208376 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:30 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208380 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.208384 | controller |       \   logger.go:42: 09:40:30 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.208388 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.208392 | controller |       \   logger.go:42: 09:40:30 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.208396 | controller |       -v '^$'\n    logger.go:42: 09:40:30 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.208403 | controller |       ++ head -1\n    logger.go:42: 09:40:30 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208407 | controller |       | + APIPOD=\n    logger.go:42: 09:40:31 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208411 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.208416 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.208426 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.208431 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.208435 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.208440 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.208444 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.208450 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.208456 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.208461 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.208465 | controller |       \   logger.go:42: 09:40:31 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.208469 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:31 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208474 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.208478 | controller |       \   logger.go:42: 09:40:31 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.208482 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.208487 | controller |       \   logger.go:42: 09:40:31 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.208491 | controller |       -1\n    logger.go:42: 09:40:31 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.208495 | controller |       -v '^$'\n    logger.go:42: 09:40:32 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.208499 | controller |       + APIPOD=\n    logger.go:42: 09:40:33 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208504 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.208508 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.208512 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.208516 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.208521 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.208526 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.208531 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.208535 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.208540 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.208544 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.208548 | controller |       \   logger.go:42: 09:40:33 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.208552 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:33 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208557 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.208561 | controller |       \   logger.go:42: 09:40:33 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.208565 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.208570 | controller |       \   logger.go:42: 09:40:33 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.208574 | controller |       -1\n    logger.go:42: 09:40:33 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.208580 | controller |       -v '^$'\n    logger.go:42: 09:40:33 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.208586 | controller |       + APIPOD=\n    logger.go:42: 09:40:34 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208592 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.208598 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.208606 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.208612 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.208632 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.208646 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.208658 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.208665 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.208672 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.208682 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.208688 | controller |       \   logger.go:42: 09:40:34 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.208693 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:34 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208699 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.208705 | controller |       \   logger.go:42: 09:40:34 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.208711 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.208716 | controller |       \   logger.go:42: 09:40:34 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.208722 | controller |       -1\n    logger.go:42: 09:40:34 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.208728 | controller |       -v '^$'\n    logger.go:42: 09:40:34 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.208734 | controller |       + APIPOD=\n    logger.go:42: 09:40:35 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208739 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.208745 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.208751 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.208757 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.208762 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.208768 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.208774 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.208780 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.208786 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.208792 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.208797 | controller |       \   logger.go:42: 09:40:35 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.208803 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:36 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208809 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.208817 | controller |       \   logger.go:42: 09:40:36 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.208823 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.208829 | controller |       \   logger.go:42: 09:40:36 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.208834 | controller |       -1\n    logger.go:42: 09:40:36 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.208840 | controller |       -v '^$'\n    logger.go:42: 09:40:36 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.208846 | controller |       + APIPOD=\n    logger.go:42: 09:40:37 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208852 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.208857 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.208863 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.208868 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.208874 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.208879 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.208894 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.208901 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.208907 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.208913 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.208919 | controller |       \   logger.go:42: 09:40:37 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.208924 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:37 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208929 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.208933 | controller |       \   logger.go:42: 09:40:37 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.208940 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.208944 | controller |       \   logger.go:42: 09:40:37 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.208948 | controller |       -1\n    logger.go:42: 09:40:37 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.208952 | controller |       -v '^$'\n    logger.go:42: 09:40:37 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.208957 | controller |       + APIPOD=\n    logger.go:42: 09:40:38 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.208961 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.208965 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.208969 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.208973 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.208977 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.208981 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.208986 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.208990 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.208996 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.209000 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.209005 | controller |       \   logger.go:42: 09:40:38 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.209009 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:38 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209013 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.209017 | controller |       \   logger.go:42: 09:40:38 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.209021 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.209025 | controller |       \   logger.go:42: 09:40:38 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.209029 | controller |       -1\n    logger.go:42: 09:40:38 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.209034 | controller |       -v '^$'\n    logger.go:42: 09:40:38 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.209038 | controller |       + APIPOD=\n    logger.go:42: 09:40:39 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209042 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.209046 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.209050 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.209054 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.209059 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.209063 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.209067 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.209080 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.209087 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.209093 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.209100 | controller |       \   logger.go:42: 09:40:39 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.209106 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:40 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209112 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.209118 | controller |       \   logger.go:42: 09:40:40 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.209122 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.209127 | controller |       \   logger.go:42: 09:40:40 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.209131 | controller |       -1\n    logger.go:42: 09:40:40 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.209135 | controller |       -v '^$'\n    logger.go:42: 09:40:40 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.209139 | controller |       + APIPOD=\n    logger.go:42: 09:40:41 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209150 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.209154 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.209159 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.209163 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.209167 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.209171 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.209176 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.209180 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.209184 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.209188 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.209193 | controller |       \   logger.go:42: 09:40:41 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.209197 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:41 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209201 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.209205 | controller |       \   logger.go:42: 09:40:41 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.209210 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.209214 | controller |       \   logger.go:42: 09:40:41 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.209218 | controller |       -1\n    logger.go:42: 09:40:41 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.209222 | controller |       -v '^$'\n    logger.go:42: 09:40:41 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.209226 | controller |       + APIPOD=\n    logger.go:42: 09:40:42 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209231 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.209235 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.209239 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.209243 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.209248 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.209252 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.209256 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.209260 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.209272 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.209279 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.209285 | controller |       \   logger.go:42: 09:40:42 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.209312 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:42 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209321 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.209327 | controller |       \   logger.go:42: 09:40:42 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.209333 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.209338 | controller |       \   logger.go:42: 09:40:42 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.209342 | controller |       -1\n    logger.go:42: 09:40:42 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.209346 | controller |       -v '^$'\n    logger.go:42: 09:40:42 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.209350 | controller |       + APIPOD=\n    logger.go:42: 09:40:44 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209355 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.209359 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.209363 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.209367 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.209371 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.209375 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.209382 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.209386 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.209390 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.209395 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.209399 | controller |       \   logger.go:42: 09:40:44 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.209403 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:44 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209407 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.209411 | controller |       \   logger.go:42: 09:40:44 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.209415 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.209419 | controller |       \   logger.go:42: 09:40:44 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.209424 | controller |       -1\n    logger.go:42: 09:40:44 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.209428 | controller |       -v '^$'\n    logger.go:42: 09:40:44 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.209432 | controller |       + APIPOD=\n    logger.go:42: 09:40:45 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209436 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.209440 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.209444 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.209449 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.209453 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.209457 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.209463 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.209467 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.209471 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.209475 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.209487 | controller |       \   logger.go:42: 09:40:45 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.209494 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:45 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209500 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.209507 | controller |       \   logger.go:42: 09:40:45 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.209513 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.209518 | controller |       \   logger.go:42: 09:40:45 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.209524 | controller |       -1\n    logger.go:42: 09:40:45 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.209530 | controller |       -v '^$'\n    logger.go:42: 09:40:46 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.209536 | controller |       + APIPOD=\n    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209542 | controller |       | running command: [sh -c set -euxo pipefail\n        oc project watcher-kuttl-default\n
2026-03-14 09:44:27.209547 | controller |       \       APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\"
2026-03-14 09:44:27.209553 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n        if [ -n \"${APIPOD}\"
2026-03-14 09:44:27.209558 | controller |       ]; then\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.209563 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.209567 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.209571 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.209576 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.209580 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.209584 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.209589 | controller |       \   logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.209596 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209600 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.209605 | controller |       \   logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.209609 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.209613 | controller |       \   logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.209619 | controller |       -1\n    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.209625 | controller |       -v '^$'\n    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.209629 | controller |       + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209633 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209638 | controller |       | ++ grep -c '^# Global config'\n    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209642 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:44:27.209646 | controller |       \   logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account | error: unable
2026-03-14 09:44:27.209650 | controller |       to upgrade connection: container not found (\"watcher-api\")\n    logger.go:42:
2026-03-14 09:44:27.209655 | controller |       09:40:47 | watcher/4-deploy-with-precreated-account | ++ echo\n    logger.go:42:
2026-03-14 09:44:27.209659 | controller |       09:40:47 | watcher/4-deploy-with-precreated-account | + '[' 0 == 1 ']'\n    logger.go:42:
2026-03-14 09:44:27.209663 | controller |       09:40:48 | watcher/4-deploy-with-precreated-account | running command: [sh -c set
2026-03-14 09:44:27.209667 | controller |       -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc get
2026-03-14 09:44:27.209672 | controller |       pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:44:27.209676 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.209680 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.209685 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.209691 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.209720 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.209728 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.209735 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.209742 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.209748 | controller |       \   logger.go:42: 09:40:48 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.209753 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:48 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209757 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.209761 | controller |       \   logger.go:42: 09:40:48 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.209766 | controller |       -v '^$'\n    logger.go:42: 09:40:48 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.209770 | controller |       ++ head -1\n    logger.go:42: 09:40:48 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209774 | controller |       | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.209779 | controller |       \   logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n
2026-03-14 09:44:27.209783 | controller |       \   logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | + '[' -n
2026-03-14 09:44:27.209787 | controller |       watcher-kuttl-api-0 ']'\n    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209791 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:44:27.209796 | controller |       \   logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.209802 | controller |       -c '^# Global config'\n    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209806 | controller |       | ++ echo '#' Global config\n    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209811 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209815 | controller |       | ++ grep -c '^# Service config'\n    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209819 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n
2026-03-14 09:44:27.209823 | controller |       \   logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.209827 | controller |       '#' Service config\n    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209832 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.209836 | controller |       | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n
2026-03-14 09:44:27.209841 | controller |       \   logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | +++ oc rsh
2026-03-14 09:44:27.209845 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.209849 | controller |       \   logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.209853 | controller |       '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.209858 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.209862 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.209866 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_test:8df261d072f382ca7aeb59823b1f461f@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.209871 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.209875 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.209879 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.209883 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.209887 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.209899 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.209906 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.209912 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.209919 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.209926 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.209933 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.209939 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.209944 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.209949 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.209955 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.209963 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.209968 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.209973 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.209977 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.209981 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]'
2026-03-14 09:44:27.209985 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.209989 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.210009 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.210013 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.210018 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.210022 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.210027 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.210031 | controller |       period = 900\n    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210035 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210039 | controller |       | ++ grep -czPo 'TimeOut 80'\n    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210043 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n
2026-03-14 09:44:27.210047 | controller |       \   logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.210052 | controller |       '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '<VirtualHost'
2026-03-14 09:44:27.210056 | controller |       '*:9322>' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot
2026-03-14 09:44:27.210060 | controller |       DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.210064 | controller |       80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.210068 | controller |       '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.210072 | controller |       AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog /dev/stdout
2026-03-14 09:44:27.210077 | controller |       ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout
2026-03-14 09:44:27.210081 | controller |       proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI
2026-03-14 09:44:27.210085 | controller |       configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal
2026-03-14 09:44:27.210089 | controller |       group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias
2026-03-14 09:44:27.210093 | controller |       / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>' '#' public vhost watcher-public.watcher-kuttl-default.svc
2026-03-14 09:44:27.210097 | controller |       configuration '<VirtualHost' '*:9322>' ServerName watcher-public.watcher-kuttl-default.svc
2026-03-14 09:44:27.210102 | controller |       '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the
2026-03-14 09:44:27.210106 | controller |       watcher-api TimeOut 80 '##' Directories, there should at least be a declaration
2026-03-14 09:44:27.210110 | controller |       for /var/www/cgi-bin '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks
2026-03-14 09:44:27.210116 | controller |       +MultiViews AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog
2026-03-14 09:44:27.210127 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded'
2026-03-14 09:44:27.210134 | controller |       CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel
2026-03-14 09:44:27.210140 | controller |       debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess
2026-03-14 09:44:27.210146 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:44:27.210152 | controller |       public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>'\n    logger.go:42:
2026-03-14 09:44:27.210158 | controller |       09:40:49 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:44:27.210162 | controller |       09:40:51 | watcher/4-deploy-with-precreated-account | running command: [sh -c set
2026-03-14 09:44:27.210167 | controller |       -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc get
2026-03-14 09:44:27.210171 | controller |       pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:44:27.210175 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.210179 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.210183 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.210187 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.210191 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.210198 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.210203 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.210207 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.210211 | controller |       \   logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.210215 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210220 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.210224 | controller |       \   logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.210228 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.210236 | controller |       \   logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.210240 | controller |       -1\n    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.210244 | controller |       -v '^$'\n    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.210249 | controller |       + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210253 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210257 | controller |       | ++ grep -c '^# Global config'\n    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210261 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:44:27.210265 | controller |       \   logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.210272 | controller |       '#' Global config\n    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210276 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210280 | controller |       | ++ grep -c '^# Service config'\n    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210284 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n
2026-03-14 09:44:27.210307 | controller |       \   logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.210317 | controller |       '#' Service config\n    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210323 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210329 | controller |       | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n
2026-03-14 09:44:27.210333 | controller |       \   logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | +++ oc rsh
2026-03-14 09:44:27.210337 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.210348 | controller |       \   logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.210355 | controller |       '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.210361 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.210368 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.210374 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_test:8df261d072f382ca7aeb59823b1f461f@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.210379 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.210388 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.210394 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.210399 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.210403 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.210407 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.210411 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.210416 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.210420 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.210424 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.210428 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.210433 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.210437 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.210441 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.210445 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.210450 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.210456 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.210460 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.210465 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.210469 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]'
2026-03-14 09:44:27.210473 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.210477 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.210482 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.210486 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.210490 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.210494 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.210499 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.210503 | controller |       period = 900\n    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210507 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210512 | controller |       | ++ grep -czPo 'TimeOut 80'\n    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210516 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n
2026-03-14 09:44:27.210520 | controller |       \   logger.go:42: 09:40:52 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.210524 | controller |       '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '<VirtualHost'
2026-03-14 09:44:27.210529 | controller |       '*:9322>' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot
2026-03-14 09:44:27.210533 | controller |       DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.210537 | controller |       80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.210548 | controller |       '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.210558 | controller |       AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog /dev/stdout
2026-03-14 09:44:27.210565 | controller |       ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout
2026-03-14 09:44:27.210571 | controller |       proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI
2026-03-14 09:44:27.210577 | controller |       configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal
2026-03-14 09:44:27.210582 | controller |       group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias
2026-03-14 09:44:27.210588 | controller |       / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>' '#' public vhost watcher-public.watcher-kuttl-default.svc
2026-03-14 09:44:27.210594 | controller |       configuration '<VirtualHost' '*:9322>' ServerName watcher-public.watcher-kuttl-default.svc
2026-03-14 09:44:27.210600 | controller |       '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the
2026-03-14 09:44:27.210606 | controller |       watcher-api TimeOut 80 '##' Directories, there should at least be a declaration
2026-03-14 09:44:27.210612 | controller |       for /var/www/cgi-bin '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks
2026-03-14 09:44:27.210617 | controller |       +MultiViews AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog
2026-03-14 09:44:27.210623 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded'
2026-03-14 09:44:27.210627 | controller |       CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel
2026-03-14 09:44:27.210632 | controller |       debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess
2026-03-14 09:44:27.210636 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:44:27.210640 | controller |       public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>'\n    logger.go:42:
2026-03-14 09:44:27.210644 | controller |       09:40:52 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:44:27.210649 | controller |       09:40:53 | watcher/4-deploy-with-precreated-account | running command: [sh -c set
2026-03-14 09:44:27.210653 | controller |       -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc get
2026-03-14 09:44:27.210657 | controller |       pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:44:27.210661 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.210666 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.210670 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.210674 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.210678 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.210683 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.210687 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.210691 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.210696 | controller |       \   logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.210700 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210704 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.210708 | controller |       \   logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.210713 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.210717 | controller |       \   logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.210721 | controller |       -1\n    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.210725 | controller |       -v '^$'\n    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.210730 | controller |       + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210734 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210738 | controller |       | ++ grep -c '^# Global config'\n    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210742 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:44:27.210753 | controller |       \   logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.210760 | controller |       '#' Global config\n    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210767 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210775 | controller |       | ++ grep -c '^# Service config'\n    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210784 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n
2026-03-14 09:44:27.210790 | controller |       \   logger.go:42: 09:40:54 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.210795 | controller |       '#' Service config\n    logger.go:42: 09:40:54 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210801 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:40:54 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210807 | controller |       | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n
2026-03-14 09:44:27.210813 | controller |       \   logger.go:42: 09:40:54 | watcher/4-deploy-with-precreated-account | +++ oc rsh
2026-03-14 09:44:27.210818 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.210824 | controller |       \   logger.go:42: 09:40:54 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.210830 | controller |       '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.210836 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.210842 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.210847 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_test:8df261d072f382ca7aeb59823b1f461f@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.210853 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.210859 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.210863 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.210868 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.210872 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.210876 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.210880 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.210886 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.210890 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.210894 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.210898 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.210902 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.210907 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.210911 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.210915 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.210919 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.210923 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.210927 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.210933 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.210937 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]'
2026-03-14 09:44:27.210942 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.210946 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.210950 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.210954 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.210965 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.210972 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.210978 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.210984 | controller |       period = 900\n    logger.go:42: 09:40:54 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210990 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:40:54 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.210996 | controller |       | ++ grep -czPo 'TimeOut 80'\n    logger.go:42: 09:40:54 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211002 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n
2026-03-14 09:44:27.211008 | controller |       \   logger.go:42: 09:40:54 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.211014 | controller |       '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '<VirtualHost'
2026-03-14 09:44:27.211019 | controller |       '*:9322>' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot
2026-03-14 09:44:27.211025 | controller |       DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.211031 | controller |       80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.211036 | controller |       '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.211042 | controller |       AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog /dev/stdout
2026-03-14 09:44:27.211048 | controller |       ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout
2026-03-14 09:44:27.211054 | controller |       proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI
2026-03-14 09:44:27.211059 | controller |       configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal
2026-03-14 09:44:27.211065 | controller |       group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias
2026-03-14 09:44:27.211071 | controller |       / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>' '#' public vhost watcher-public.watcher-kuttl-default.svc
2026-03-14 09:44:27.211077 | controller |       configuration '<VirtualHost' '*:9322>' ServerName watcher-public.watcher-kuttl-default.svc
2026-03-14 09:44:27.211082 | controller |       '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the
2026-03-14 09:44:27.211088 | controller |       watcher-api TimeOut 80 '##' Directories, there should at least be a declaration
2026-03-14 09:44:27.211094 | controller |       for /var/www/cgi-bin '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks
2026-03-14 09:44:27.211100 | controller |       +MultiViews AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog
2026-03-14 09:44:27.211105 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded'
2026-03-14 09:44:27.211111 | controller |       CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel
2026-03-14 09:44:27.211119 | controller |       debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess
2026-03-14 09:44:27.211125 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:44:27.211131 | controller |       public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>'\n    logger.go:42:
2026-03-14 09:44:27.211136 | controller |       09:40:54 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:44:27.211142 | controller |       09:40:55 | watcher/4-deploy-with-precreated-account | running command: [sh -c set
2026-03-14 09:44:27.211148 | controller |       -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc get
2026-03-14 09:44:27.211154 | controller |       pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:44:27.211159 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.211163 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.211167 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.211173 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.211177 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.211182 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.211186 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.211190 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.211202 | controller |       \   logger.go:42: 09:40:55 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.211208 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:55 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211214 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.211221 | controller |       \   logger.go:42: 09:40:55 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.211227 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.211232 | controller |       \   logger.go:42: 09:40:55 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.211236 | controller |       -1\n    logger.go:42: 09:40:55 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.211240 | controller |       -v '^$'\n    logger.go:42: 09:40:55 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.211247 | controller |       + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:40:55 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211251 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:40:55 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211255 | controller |       | ++ grep -c '^# Global config'\n    logger.go:42: 09:40:55 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211260 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:44:27.211264 | controller |       \   logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.211268 | controller |       '#' Global config\n    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211272 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211276 | controller |       | ++ grep -c '^# Service config'\n    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211281 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n
2026-03-14 09:44:27.211302 | controller |       \   logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.211310 | controller |       '#' Service config\n    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211314 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211318 | controller |       | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n
2026-03-14 09:44:27.211322 | controller |       \   logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account | +++ oc rsh
2026-03-14 09:44:27.211327 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.211331 | controller |       \   logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.211335 | controller |       '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.211339 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.211343 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.211348 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_test:8df261d072f382ca7aeb59823b1f461f@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.211352 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.211356 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.211360 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.211365 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.211369 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.211373 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.211377 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.211382 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.211386 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.211390 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.211394 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.211406 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.211413 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.211419 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.211426 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.211432 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.211438 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.211444 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.211448 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.211455 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]'
2026-03-14 09:44:27.211459 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.211464 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.211468 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.211472 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.211476 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.211481 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.211485 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.211489 | controller |       period = 900\n    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211494 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211498 | controller |       | ++ grep -czPo 'TimeOut 80'\n    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211506 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n
2026-03-14 09:44:27.211510 | controller |       \   logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.211515 | controller |       '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '<VirtualHost'
2026-03-14 09:44:27.211519 | controller |       '*:9322>' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot
2026-03-14 09:44:27.211523 | controller |       DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.211528 | controller |       80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.211533 | controller |       '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.211537 | controller |       AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog /dev/stdout
2026-03-14 09:44:27.211541 | controller |       ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout
2026-03-14 09:44:27.211546 | controller |       proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI
2026-03-14 09:44:27.211550 | controller |       configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal
2026-03-14 09:44:27.211556 | controller |       group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias
2026-03-14 09:44:27.211560 | controller |       / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>' '#' public vhost watcher-public.watcher-kuttl-default.svc
2026-03-14 09:44:27.211564 | controller |       configuration '<VirtualHost' '*:9322>' ServerName watcher-public.watcher-kuttl-default.svc
2026-03-14 09:44:27.211569 | controller |       '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the
2026-03-14 09:44:27.211573 | controller |       watcher-api TimeOut 80 '##' Directories, there should at least be a declaration
2026-03-14 09:44:27.211577 | controller |       for /var/www/cgi-bin '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks
2026-03-14 09:44:27.211581 | controller |       +MultiViews AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog
2026-03-14 09:44:27.211585 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded'
2026-03-14 09:44:27.211590 | controller |       CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel
2026-03-14 09:44:27.211594 | controller |       debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess
2026-03-14 09:44:27.211598 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:44:27.211604 | controller |       public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>'\n    logger.go:42:
2026-03-14 09:44:27.211608 | controller |       09:40:56 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:44:27.211620 | controller |       09:40:57 | watcher/4-deploy-with-precreated-account | running command: [sh -c set
2026-03-14 09:44:27.211627 | controller |       -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc get
2026-03-14 09:44:27.211634 | controller |       pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:44:27.211643 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.211649 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.211654 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.211660 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.211666 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.211671 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.211676 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.211680 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.211684 | controller |       \   logger.go:42: 09:40:57 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.211688 | controller |       watcher-kuttl-default\n    logger.go:42: 09:40:57 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211693 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.211697 | controller |       \   logger.go:42: 09:40:57 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.211701 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.211705 | controller |       \   logger.go:42: 09:40:57 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.211710 | controller |       -1\n    logger.go:42: 09:40:57 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.211714 | controller |       -v '^$'\n    logger.go:42: 09:40:57 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.211718 | controller |       + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:40:57 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211722 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:40:57 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211727 | controller |       | ++ grep -c '^# Global config'\n    logger.go:42: 09:40:57 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211731 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:44:27.211735 | controller |       \   logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.211740 | controller |       '#' Global config\n    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211744 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211748 | controller |       | ++ grep -c '^# Service config'\n    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211756 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n
2026-03-14 09:44:27.211760 | controller |       \   logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.211765 | controller |       '#' Service config\n    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211771 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211775 | controller |       | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n
2026-03-14 09:44:27.211779 | controller |       \   logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account | +++ oc rsh
2026-03-14 09:44:27.211783 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.211788 | controller |       \   logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.211793 | controller |       '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.211798 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.211802 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.211815 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_test:8df261d072f382ca7aeb59823b1f461f@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.211822 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.211829 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.211835 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.211841 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.211847 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.211853 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.211859 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.211865 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.211870 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.211876 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.211882 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.211886 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.211891 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.211895 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.211901 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.211907 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.211913 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.211919 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.211925 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.211931 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]'
2026-03-14 09:44:27.211936 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.211944 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.211949 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.211955 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.211961 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.211966 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.211972 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.211977 | controller |       period = 900\n    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211983 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211988 | controller |       | ++ grep -czPo 'TimeOut 80'\n    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.211994 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n
2026-03-14 09:44:27.212000 | controller |       \   logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.212006 | controller |       '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '<VirtualHost'
2026-03-14 09:44:27.212011 | controller |       '*:9322>' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot
2026-03-14 09:44:27.212018 | controller |       DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.212024 | controller |       80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.212031 | controller |       '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.212037 | controller |       AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog /dev/stdout
2026-03-14 09:44:27.212044 | controller |       ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout
2026-03-14 09:44:27.212051 | controller |       proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI
2026-03-14 09:44:27.212057 | controller |       configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal
2026-03-14 09:44:27.212066 | controller |       group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias
2026-03-14 09:44:27.212082 | controller |       / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>' '#' public vhost watcher-public.watcher-kuttl-default.svc
2026-03-14 09:44:27.212090 | controller |       configuration '<VirtualHost' '*:9322>' ServerName watcher-public.watcher-kuttl-default.svc
2026-03-14 09:44:27.212095 | controller |       '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the
2026-03-14 09:44:27.212101 | controller |       watcher-api TimeOut 80 '##' Directories, there should at least be a declaration
2026-03-14 09:44:27.212106 | controller |       for /var/www/cgi-bin '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks
2026-03-14 09:44:27.212112 | controller |       +MultiViews AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog
2026-03-14 09:44:27.212119 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded'
2026-03-14 09:44:27.212125 | controller |       CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel
2026-03-14 09:44:27.212132 | controller |       debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess
2026-03-14 09:44:27.212139 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:44:27.212145 | controller |       public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>'\n    logger.go:42:
2026-03-14 09:44:27.212152 | controller |       09:40:58 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:44:27.212163 | controller |       09:41:00 | watcher/4-deploy-with-precreated-account | running command: [sh -c set
2026-03-14 09:44:27.212169 | controller |       -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc get
2026-03-14 09:44:27.212175 | controller |       pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:44:27.212181 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.212187 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.212194 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.212200 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.212207 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.212214 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.212221 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.212227 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.212233 | controller |       \   logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.212239 | controller |       watcher-kuttl-default\n    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212245 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.212252 | controller |       \   logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.212258 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.212265 | controller |       \   logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.212272 | controller |       -1\n    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.212278 | controller |       -v '^$'\n    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.212285 | controller |       + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212317 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212324 | controller |       | ++ grep -c '^# Global config'\n    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212331 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:44:27.212338 | controller |       \   logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.212344 | controller |       '#' Global config\n    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212351 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212357 | controller |       | ++ grep -c '^# Service config'\n    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212364 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n
2026-03-14 09:44:27.212371 | controller |       \   logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.212387 | controller |       '#' Service config\n    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212395 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212404 | controller |       | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n
2026-03-14 09:44:27.212410 | controller |       \   logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | +++ oc rsh
2026-03-14 09:44:27.212416 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.212422 | controller |       \   logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.212427 | controller |       '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.212433 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.212438 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.212444 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_test:8df261d072f382ca7aeb59823b1f461f@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.212450 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.212456 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.212460 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.212464 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.212469 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.212473 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.212477 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.212481 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.212486 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.212490 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.212494 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.212498 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.212503 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.212507 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.212511 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.212516 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.212523 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.212527 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.212532 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.212536 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]'
2026-03-14 09:44:27.212542 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.212546 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.212550 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.212557 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.212562 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.212566 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.212570 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.212575 | controller |       period = 900\n    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212579 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212590 | controller |       | ++ grep -czPo 'TimeOut 80'\n    logger.go:42: 09:41:01 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212595 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n
2026-03-14 09:44:27.212599 | controller |       \   logger.go:42: 09:41:01 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.212609 | controller |       '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '<VirtualHost'
2026-03-14 09:44:27.212614 | controller |       '*:9322>' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot
2026-03-14 09:44:27.212618 | controller |       DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.212622 | controller |       80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.212626 | controller |       '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.212630 | controller |       AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog /dev/stdout
2026-03-14 09:44:27.212635 | controller |       ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout
2026-03-14 09:44:27.212639 | controller |       proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI
2026-03-14 09:44:27.212643 | controller |       configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal
2026-03-14 09:44:27.212647 | controller |       group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias
2026-03-14 09:44:27.212651 | controller |       / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>' '#' public vhost watcher-public.watcher-kuttl-default.svc
2026-03-14 09:44:27.212655 | controller |       configuration '<VirtualHost' '*:9322>' ServerName watcher-public.watcher-kuttl-default.svc
2026-03-14 09:44:27.212660 | controller |       '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the
2026-03-14 09:44:27.212664 | controller |       watcher-api TimeOut 80 '##' Directories, there should at least be a declaration
2026-03-14 09:44:27.212668 | controller |       for /var/www/cgi-bin '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks
2026-03-14 09:44:27.212672 | controller |       +MultiViews AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog
2026-03-14 09:44:27.212676 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded'
2026-03-14 09:44:27.212681 | controller |       CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel
2026-03-14 09:44:27.212685 | controller |       debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess
2026-03-14 09:44:27.212689 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:44:27.212693 | controller |       public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>'\n    logger.go:42:
2026-03-14 09:44:27.212697 | controller |       09:41:01 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:44:27.212701 | controller |       09:41:02 | watcher/4-deploy-with-precreated-account | running command: [sh -c set
2026-03-14 09:44:27.212705 | controller |       -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc get
2026-03-14 09:44:27.212711 | controller |       pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:44:27.212715 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.212720 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.212724 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.212728 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.212732 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.212736 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.212741 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.212745 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.212749 | controller |       \   logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.212753 | controller |       watcher-kuttl-default\n    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212758 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.212762 | controller |       \   logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.212766 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.212770 | controller |       \   logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.212780 | controller |       -1\n    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.212784 | controller |       -v '^$'\n    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.212788 | controller |       + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212793 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212797 | controller |       | ++ grep -c '^# Global config'\n    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212801 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:44:27.212807 | controller |       \   logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.212811 | controller |       '#' Global config\n    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212815 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212819 | controller |       | ++ grep -c '^# Service config'\n    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212823 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n
2026-03-14 09:44:27.212828 | controller |       \   logger.go:42: 09:41:03 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.212832 | controller |       '#' Service config\n    logger.go:42: 09:41:03 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212836 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:41:03 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.212840 | controller |       | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n
2026-03-14 09:44:27.212846 | controller |       \   logger.go:42: 09:41:03 | watcher/4-deploy-with-precreated-account | +++ oc rsh
2026-03-14 09:44:27.212850 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.212854 | controller |       \   logger.go:42: 09:41:03 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.212859 | controller |       '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.212864 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.212869 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.212873 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_test:8df261d072f382ca7aeb59823b1f461f@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.212877 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.212881 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.212885 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.212890 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.212894 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.212898 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.212902 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.212906 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.212910 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.212915 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.212920 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.212925 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.212930 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.212934 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.212939 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.212945 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.212949 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.212958 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.212963 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.212967 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]'
2026-03-14 09:44:27.212971 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.212975 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.212980 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.212984 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.212988 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.212993 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.212998 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.213002 | controller |       period = 900\n    logger.go:42: 09:41:03 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213006 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:41:03 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213010 | controller |       | ++ grep -czPo 'TimeOut 80'\n    logger.go:42: 09:41:03 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213014 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n
2026-03-14 09:44:27.213019 | controller |       \   logger.go:42: 09:41:03 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.213023 | controller |       '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '<VirtualHost'
2026-03-14 09:44:27.213027 | controller |       '*:9322>' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot
2026-03-14 09:44:27.213031 | controller |       DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.213035 | controller |       80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.213039 | controller |       '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.213044 | controller |       AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog /dev/stdout
2026-03-14 09:44:27.213048 | controller |       ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout
2026-03-14 09:44:27.213052 | controller |       proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI
2026-03-14 09:44:27.213056 | controller |       configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal
2026-03-14 09:44:27.213060 | controller |       group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias
2026-03-14 09:44:27.213065 | controller |       / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>' '#' public vhost watcher-public.watcher-kuttl-default.svc
2026-03-14 09:44:27.213069 | controller |       configuration '<VirtualHost' '*:9322>' ServerName watcher-public.watcher-kuttl-default.svc
2026-03-14 09:44:27.213073 | controller |       '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the
2026-03-14 09:44:27.213079 | controller |       watcher-api TimeOut 80 '##' Directories, there should at least be a declaration
2026-03-14 09:44:27.213083 | controller |       for /var/www/cgi-bin '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks
2026-03-14 09:44:27.213087 | controller |       +MultiViews AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog
2026-03-14 09:44:27.213091 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded'
2026-03-14 09:44:27.213095 | controller |       CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel
2026-03-14 09:44:27.213099 | controller |       debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess
2026-03-14 09:44:27.213104 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:44:27.213108 | controller |       public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>'\n    logger.go:42:
2026-03-14 09:44:27.213112 | controller |       09:41:03 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:44:27.213116 | controller |       09:41:04 | watcher/4-deploy-with-precreated-account | running command: [sh -c set
2026-03-14 09:44:27.213120 | controller |       -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc get
2026-03-14 09:44:27.213125 | controller |       pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:44:27.213130 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.213135 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.213139 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.213149 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.213153 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.213157 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.213162 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.213166 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.213170 | controller |       \   logger.go:42: 09:41:04 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.213174 | controller |       watcher-kuttl-default\n    logger.go:42: 09:41:04 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213178 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.213183 | controller |       \   logger.go:42: 09:41:04 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.213187 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.213191 | controller |       \   logger.go:42: 09:41:04 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.213195 | controller |       -1\n    logger.go:42: 09:41:04 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.213199 | controller |       -v '^$'\n    logger.go:42: 09:41:04 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.213203 | controller |       + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:41:04 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213208 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:41:04 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213212 | controller |       | ++ grep -c '^# Global config'\n    logger.go:42: 09:41:04 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213216 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:44:27.213220 | controller |       \   logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.213224 | controller |       '#' Global config\n    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213229 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213233 | controller |       | ++ grep -c '^# Service config'\n    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213237 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n
2026-03-14 09:44:27.213241 | controller |       \   logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.213245 | controller |       '#' Service config\n    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213249 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213255 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.213260 | controller |       \   logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.213265 | controller |       -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n
2026-03-14 09:44:27.213271 | controller |       \   logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.213275 | controller |       '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.213279 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.213284 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.213302 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_test:8df261d072f382ca7aeb59823b1f461f@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.213309 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.213314 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.213318 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.213323 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.213327 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.213337 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.213342 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.213346 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.213350 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.213355 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.213359 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.213363 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.213367 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.213374 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.213378 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.213382 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.213386 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.213391 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.213395 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.213399 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]'
2026-03-14 09:44:27.213404 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.213408 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.213412 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.213416 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.213420 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.213425 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.213432 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.213436 | controller |       period = 900\n    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213440 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213444 | controller |       | ++ grep -czPo 'TimeOut 80'\n    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213449 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n
2026-03-14 09:44:27.213453 | controller |       \   logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.213457 | controller |       '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '<VirtualHost'
2026-03-14 09:44:27.213461 | controller |       '*:9322>' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot
2026-03-14 09:44:27.213466 | controller |       DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.213470 | controller |       80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.213474 | controller |       '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.213479 | controller |       AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog /dev/stdout
2026-03-14 09:44:27.213483 | controller |       ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout
2026-03-14 09:44:27.213487 | controller |       proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI
2026-03-14 09:44:27.213491 | controller |       configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal
2026-03-14 09:44:27.213496 | controller |       group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias
2026-03-14 09:44:27.213500 | controller |       / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>' '#' public vhost watcher-public.watcher-kuttl-default.svc
2026-03-14 09:44:27.213504 | controller |       configuration '<VirtualHost' '*:9322>' ServerName watcher-public.watcher-kuttl-default.svc
2026-03-14 09:44:27.213508 | controller |       '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the
2026-03-14 09:44:27.213512 | controller |       watcher-api TimeOut 80 '##' Directories, there should at least be a declaration
2026-03-14 09:44:27.213517 | controller |       for /var/www/cgi-bin '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks
2026-03-14 09:44:27.213527 | controller |       +MultiViews AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog
2026-03-14 09:44:27.213531 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded'
2026-03-14 09:44:27.213536 | controller |       CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel
2026-03-14 09:44:27.213540 | controller |       debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess
2026-03-14 09:44:27.213544 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:44:27.213549 | controller |       public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>'\n    logger.go:42:
2026-03-14 09:44:27.213553 | controller |       09:41:05 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:44:27.213558 | controller |       09:41:07 | watcher/4-deploy-with-precreated-account | running command: [sh -c set
2026-03-14 09:44:27.213563 | controller |       -euxo pipefail\n        oc project watcher-kuttl-default\n        APIPOD=$(oc get
2026-03-14 09:44:27.213568 | controller |       pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep
2026-03-14 09:44:27.213572 | controller |       -v ^$|head -1)\n        if [ -n \"${APIPOD}\" ]; then\n          [ $(echo $(oc rsh
2026-03-14 09:44:27.213576 | controller |       -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.213583 | controller |       |grep -c \"^# Global config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api
2026-03-14 09:44:27.213587 | controller |       ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^#
2026-03-14 09:44:27.213591 | controller |       Service config\") == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat
2026-03-14 09:44:27.213596 | controller |       /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem')
2026-03-14 09:44:27.213600 | controller |       == 1 ]\n          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.213604 | controller |       |grep -czPo 'TimeOut 80') == 1 ]\n        else\n          exit 1\n        fi\n        ]\n
2026-03-14 09:44:27.213608 | controller |       \   logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.213613 | controller |       watcher-kuttl-default\n    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213617 | controller |       | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n
2026-03-14 09:44:27.213621 | controller |       \   logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.213627 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n
2026-03-14 09:44:27.213631 | controller |       \   logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.213635 | controller |       -1\n    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.213639 | controller |       -v '^$'\n    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account |
2026-03-14 09:44:27.213643 | controller |       + APIPOD=watcher-kuttl-api-0\n    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213648 | controller |       | + '[' -n watcher-kuttl-api-0 ']'\n    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213652 | controller |       | ++ grep -c '^# Global config'\n    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213658 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n
2026-03-14 09:44:27.213662 | controller |       \   logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.213666 | controller |       '#' Global config\n    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213670 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213675 | controller |       | ++ grep -c '^# Service config'\n    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213679 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n
2026-03-14 09:44:27.213683 | controller |       \   logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.213687 | controller |       '#' Service config\n    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213691 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213696 | controller |       | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n
2026-03-14 09:44:27.213700 | controller |       \   logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | +++ oc rsh
2026-03-14 09:44:27.213726 | controller |       -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n
2026-03-14 09:44:27.213733 | controller |       \   logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.213739 | controller |       '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'
2026-03-14 09:44:27.213746 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.213752 | controller |       '#' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.213757 | controller |       = '[database]' connection = 'mysql+pymysql://watcher_test:8df261d072f382ca7aeb59823b1f461f@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'
2026-03-14 09:44:27.213762 | controller |       '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]'
2026-03-14 09:44:27.213767 | controller |       driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.213772 | controller |       amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.213777 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.213782 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.213787 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.213792 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.213797 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.213802 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.213806 | controller |       '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name
2026-03-14 09:44:27.213811 | controller |       = Default password = password username = watcher auth_type = password auth_url =
2026-03-14 09:44:27.213816 | controller |       https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile
2026-03-14 09:44:27.213821 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path
2026-03-14 09:44:27.213826 | controller |       = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]'
2026-03-14 09:44:27.213831 | controller |       backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.213836 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.213841 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.213846 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]'
2026-03-14 09:44:27.213851 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.213856 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]'
2026-03-14 09:44:27.213860 | controller |       endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type
2026-03-14 09:44:27.213865 | controller |       = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.213870 | controller |       = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]'
2026-03-14 09:44:27.213875 | controller |       interface = internal region_name = regionOne '[neutron_client]' endpoint_type =
2026-03-14 09:44:27.213880 | controller |       internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name
2026-03-14 09:44:27.213885 | controller |       = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]'
2026-03-14 09:44:27.213890 | controller |       period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]'
2026-03-14 09:44:27.213895 | controller |       period = 900\n    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213902 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213907 | controller |       | ++ grep -czPo 'TimeOut 80'\n    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account
2026-03-14 09:44:27.213912 | controller |       | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n
2026-03-14 09:44:27.213916 | controller |       \   logger.go:42: 09:41:08 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.213921 | controller |       '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '<VirtualHost'
2026-03-14 09:44:27.213926 | controller |       '*:9322>' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot
2026-03-14 09:44:27.213931 | controller |       DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.213941 | controller |       80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.213948 | controller |       '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.213953 | controller |       AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog /dev/stdout
2026-03-14 09:44:27.213958 | controller |       ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout
2026-03-14 09:44:27.213963 | controller |       proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI
2026-03-14 09:44:27.213968 | controller |       configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal
2026-03-14 09:44:27.213972 | controller |       group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias
2026-03-14 09:44:27.213976 | controller |       / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>' '#' public vhost watcher-public.watcher-kuttl-default.svc
2026-03-14 09:44:27.213980 | controller |       configuration '<VirtualHost' '*:9322>' ServerName watcher-public.watcher-kuttl-default.svc
2026-03-14 09:44:27.213984 | controller |       '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the
2026-03-14 09:44:27.213989 | controller |       watcher-api TimeOut 80 '##' Directories, there should at least be a declaration
2026-03-14 09:44:27.213993 | controller |       for /var/www/cgi-bin '<Directory' '\"/var/www/cgi-bin\">' Options -Indexes +FollowSymLinks
2026-03-14 09:44:27.213997 | controller |       +MultiViews AllowOverride None Require all granted '</Directory>' '##' Logging ErrorLog
2026-03-14 09:44:27.214001 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded'
2026-03-14 09:44:27.214005 | controller |       CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel
2026-03-14 09:44:27.214010 | controller |       debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess
2026-03-14 09:44:27.214014 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:44:27.214018 | controller |       public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' '</VirtualHost>'\n    logger.go:42:
2026-03-14 09:44:27.214022 | controller |       09:41:08 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n    logger.go:42:
2026-03-14 09:44:27.214026 | controller |       09:41:08 | watcher/4-deploy-with-precreated-account | test step completed 4-deploy-with-precreated-account\n
2026-03-14 09:44:27.214030 | controller |       \   logger.go:42: 09:41:08 | watcher/5-cleanup-watcher | starting test step 5-cleanup-watcher\n
2026-03-14 09:44:27.214035 | controller |       \   logger.go:42: 09:41:08 | watcher/5-cleanup-watcher | running command: [sh -c
2026-03-14 09:44:27.214039 | controller |       set -ex\n        oc exec -n watcher-kuttl-default openstackclient -- openstack service
2026-03-14 09:44:27.214045 | controller |       list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.214049 | controller |       09:41:08 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.214053 | controller |       -- openstack service list -f value -c Name -c Type\n    logger.go:42: 09:41:08 |
2026-03-14 09:44:27.214059 | controller |       watcher/5-cleanup-watcher | ++ grep -c '^watcher'\n    logger.go:42: 09:41:11 |
2026-03-14 09:44:27.214063 | controller |       watcher/5-cleanup-watcher | + '[' 1 == 0 ']'\n    logger.go:42: 09:41:13 | watcher/5-cleanup-watcher
2026-03-14 09:44:27.214067 | controller |       | running command: [sh -c set -ex\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.214071 | controller |       -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0
2026-03-14 09:44:27.214075 | controller |       ]\n        ]\n    logger.go:42: 09:41:13 | watcher/5-cleanup-watcher | + oc exec
2026-03-14 09:44:27.214080 | controller |       -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.214084 | controller |       -c Type\n    logger.go:42: 09:41:13 | watcher/5-cleanup-watcher | ++ grep -c '^watcher'\n
2026-03-14 09:44:27.214088 | controller |       \   logger.go:42: 09:41:15 | watcher/5-cleanup-watcher | + '[' 0 == 0 ']'\n    logger.go:42:
2026-03-14 09:44:27.214092 | controller |       09:41:16 | watcher/5-cleanup-watcher | running command: [sh -c set -ex\n        oc
2026-03-14 09:44:27.214096 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.214101 | controller |       -c Name -c Type | [ $(grep -c ^watcher) == 0 ]\n        ]\n    logger.go:42: 09:41:16
2026-03-14 09:44:27.214106 | controller |       | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.214110 | controller |       -- openstack service list -f value -c Name -c Type\n    logger.go:42: 09:41:16 |
2026-03-14 09:44:27.214114 | controller |       watcher/5-cleanup-watcher | ++ grep -c '^watcher'\n    logger.go:42: 09:41:18 |
2026-03-14 09:44:27.214118 | controller |       watcher/5-cleanup-watcher | + '[' 0 == 0 ']'\n    logger.go:42: 09:41:18 | watcher/5-cleanup-watcher
2026-03-14 09:44:27.214122 | controller |       | test step completed 5-cleanup-watcher\n    logger.go:42: 09:41:18 | watcher |
2026-03-14 09:44:27.214126 | controller |       skipping kubernetes event logging\n=== CONT  kuttl/harness/watcher-cinder\n    logger.go:42:
2026-03-14 09:44:27.214135 | controller |       09:41:18 | watcher-cinder | Skipping creation of user-supplied namespace: watcher-kuttl-default\n
2026-03-14 09:44:27.214139 | controller |       \   logger.go:42: 09:41:18 | watcher-cinder/0-cleanup-watcher | starting test step
2026-03-14 09:44:27.214143 | controller |       0-cleanup-watcher\n    logger.go:42: 09:41:18 | watcher-cinder/0-cleanup-watcher
2026-03-14 09:44:27.214148 | controller |       | test step completed 0-cleanup-watcher\n    logger.go:42: 09:41:18 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214152 | controller |       | starting test step 1-deploy-watcher-no-cinder\n    logger.go:42: 09:41:18 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214156 | controller |       | Watcher:watcher-kuttl-default/watcher-kuttl created\n    logger.go:42: 09:41:18
2026-03-14 09:44:27.214160 | controller |       | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo
2026-03-14 09:44:27.214164 | controller |       pipefail\n        # check that the decision engine correctly detects that there
2026-03-14 09:44:27.214168 | controller |       is no cinder service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.214173 | controller |       | grep -c 'Block storage service is not enabled, skipping storage collector')\"
2026-03-14 09:44:27.214177 | controller |       == 2 ]\n        ]\n    logger.go:42: 09:41:18 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214181 | controller |       | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.214185 | controller |       09:41:18 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage
2026-03-14 09:44:27.214191 | controller |       service is not enabled, skipping storage collector'\n    logger.go:42: 09:41:18
2026-03-14 09:44:27.214196 | controller |       | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound):
2026-03-14 09:44:27.214200 | controller |       pods \"watcher-kuttl-decision-engine-0\" not found in namespace \"watcher-kuttl-default\"\n
2026-03-14 09:44:27.214206 | controller |       \   logger.go:42: 09:41:18 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0
2026-03-14 09:44:27.214210 | controller |       == 2 ']'\n    logger.go:42: 09:41:19 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214214 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.214218 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.214222 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.214227 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.214231 | controller |       09:41:19 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage
2026-03-14 09:44:27.214235 | controller |       service is not enabled, skipping storage collector'\n    logger.go:42: 09:41:19
2026-03-14 09:44:27.214239 | controller |       | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.214243 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:20 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214247 | controller |       | error: error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\"
2026-03-14 09:44:27.214252 | controller |       not found in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:41:20 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214256 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:41:21 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214260 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.214264 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.214268 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.214272 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.214276 | controller |       09:41:21 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.214281 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:21 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214285 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.214301 | controller |       \   logger.go:42: 09:41:21 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.214307 | controller |       error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found
2026-03-14 09:44:27.214312 | controller |       in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:41:21 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214316 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:41:22 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214320 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.214325 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.214329 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.214333 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.214337 | controller |       09:41:22 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage
2026-03-14 09:44:27.214346 | controller |       service is not enabled, skipping storage collector'\n    logger.go:42: 09:41:22
2026-03-14 09:44:27.214351 | controller |       | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.214355 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:22 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214359 | controller |       | error: error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\"
2026-03-14 09:44:27.214363 | controller |       not found in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:41:22 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214369 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:41:23 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214373 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.214377 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.214381 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.214386 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.214390 | controller |       09:41:23 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.214394 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:23 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214400 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.214404 | controller |       \   logger.go:42: 09:41:23 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.214408 | controller |       error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found
2026-03-14 09:44:27.214412 | controller |       in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:41:23 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214417 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:41:24 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214421 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.214425 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.214429 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.214433 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.214437 | controller |       09:41:24 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage
2026-03-14 09:44:27.214442 | controller |       service is not enabled, skipping storage collector'\n    logger.go:42: 09:41:24
2026-03-14 09:44:27.214446 | controller |       | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.214450 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:24 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214454 | controller |       | error: error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\"
2026-03-14 09:44:27.214458 | controller |       not found in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:41:24 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214462 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:41:25 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214467 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.214471 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.214475 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.214479 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.214483 | controller |       09:41:25 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.214488 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:25 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214492 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.214496 | controller |       \   logger.go:42: 09:41:25 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.214500 | controller |       error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found
2026-03-14 09:44:27.214504 | controller |       in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:41:25 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214510 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:41:26 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214516 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.214520 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.214524 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.214528 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.214537 | controller |       09:41:26 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage
2026-03-14 09:44:27.214542 | controller |       service is not enabled, skipping storage collector'\n    logger.go:42: 09:41:26
2026-03-14 09:44:27.214546 | controller |       | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.214550 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:27 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214555 | controller |       | error: error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\"
2026-03-14 09:44:27.214559 | controller |       not found in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:41:27 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214564 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:41:28 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214569 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.214573 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.214577 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.214581 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.214585 | controller |       09:41:28 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.214590 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:28 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214594 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.214601 | controller |       \   logger.go:42: 09:41:28 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.214605 | controller |       error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found
2026-03-14 09:44:27.214609 | controller |       in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:41:28 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214613 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:41:29 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214617 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.214622 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.214626 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.214630 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.214634 | controller |       09:41:29 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.214638 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:29 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214642 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.214647 | controller |       \   logger.go:42: 09:41:29 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.214651 | controller |       error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found
2026-03-14 09:44:27.214655 | controller |       in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:41:29 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214659 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:41:30 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214665 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.214669 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.214673 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.214677 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.214681 | controller |       09:41:30 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage
2026-03-14 09:44:27.214685 | controller |       service is not enabled, skipping storage collector'\n    logger.go:42: 09:41:30
2026-03-14 09:44:27.214690 | controller |       | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.214695 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:30 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214699 | controller |       | error: error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\"
2026-03-14 09:44:27.214703 | controller |       not found in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:41:30 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214707 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:41:31 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214712 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.214718 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.214734 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.214743 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.214759 | controller |       09:41:31 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.214766 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:31 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214773 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.214779 | controller |       \   logger.go:42: 09:41:31 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.214785 | controller |       error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found
2026-03-14 09:44:27.214789 | controller |       in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:41:31 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214794 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:41:32 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214798 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.214802 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.214807 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.214811 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.214817 | controller |       09:41:32 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.214823 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:32 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214829 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.214835 | controller |       \   logger.go:42: 09:41:32 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.214841 | controller |       error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found
2026-03-14 09:44:27.214847 | controller |       in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:41:32 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214853 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:41:33 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214867 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.214874 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.214880 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.214885 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.214891 | controller |       09:41:33 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.214897 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:33 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214903 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.214909 | controller |       \   logger.go:42: 09:41:33 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.214915 | controller |       error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found
2026-03-14 09:44:27.214921 | controller |       in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:41:33 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214927 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:41:34 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214932 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.214936 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.214941 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.214945 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.214949 | controller |       09:41:34 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.214953 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:34 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214957 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.214961 | controller |       \   logger.go:42: 09:41:35 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.214966 | controller |       error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found
2026-03-14 09:44:27.214970 | controller |       in namespace \"watcher-kuttl-default\"\n    logger.go:42: 09:41:35 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214974 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:41:36 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.214978 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.214982 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.214986 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.214991 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.215003 | controller |       09:41:36 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.215008 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:36 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215012 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.215016 | controller |       \   logger.go:42: 09:41:36 | watcher-cinder/1-deploy-watcher-no-cinder | Error from
2026-03-14 09:44:27.215021 | controller |       server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\"
2026-03-14 09:44:27.215025 | controller |       is waiting to start: ContainerCreating\n    logger.go:42: 09:41:36 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215029 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:41:37 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215037 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215041 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.215045 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.215049 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.215053 | controller |       09:41:37 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.215057 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:37 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215061 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.215066 | controller |       \   logger.go:42: 09:41:37 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0
2026-03-14 09:44:27.215070 | controller |       == 2 ']'\n    logger.go:42: 09:41:38 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215076 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215081 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.215087 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.215093 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.215102 | controller |       09:41:38 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage
2026-03-14 09:44:27.215108 | controller |       service is not enabled, skipping storage collector'\n    logger.go:42: 09:41:38
2026-03-14 09:44:27.215114 | controller |       | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.215120 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:38 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215127 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:41:39 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215134 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215139 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.215145 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.215151 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.215156 | controller |       09:41:39 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.215162 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:39 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215168 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.215174 | controller |       \   logger.go:42: 09:41:40 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 1
2026-03-14 09:44:27.215180 | controller |       == 2 ']'\n    logger.go:42: 09:41:41 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215186 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215192 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.215197 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.215203 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.215208 | controller |       09:41:41 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.215214 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:41 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215220 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.215228 | controller |       \   logger.go:42: 09:41:41 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2
2026-03-14 09:44:27.215234 | controller |       == 2 ']'\n    logger.go:42: 09:41:42 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215240 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215255 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.215263 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.215269 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.215274 | controller |       09:41:42 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.215280 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:42 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215301 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.215310 | controller |       \   logger.go:42: 09:41:42 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2
2026-03-14 09:44:27.215315 | controller |       == 2 ']'\n    logger.go:42: 09:41:43 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215319 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215323 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.215327 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.215331 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.215336 | controller |       09:41:43 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.215340 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:43 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215344 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.215348 | controller |       \   logger.go:42: 09:41:43 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2
2026-03-14 09:44:27.215352 | controller |       == 2 ']'\n    logger.go:42: 09:41:44 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215359 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215364 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.215368 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.215372 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.215376 | controller |       09:41:44 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.215380 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:44 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215384 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.215388 | controller |       \   logger.go:42: 09:41:44 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2
2026-03-14 09:44:27.215393 | controller |       == 2 ']'\n    logger.go:42: 09:41:45 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215397 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215401 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.215405 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.215409 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.215413 | controller |       09:41:45 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.215417 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:45 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215424 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.215428 | controller |       \   logger.go:42: 09:41:45 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2
2026-03-14 09:44:27.215432 | controller |       == 2 ']'\n    logger.go:42: 09:41:47 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215436 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215440 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.215444 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is
2026-03-14 09:44:27.215449 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.215453 | controller |       09:41:47 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage
2026-03-14 09:44:27.215457 | controller |       service is not enabled, skipping storage collector'\n    logger.go:42: 09:41:47
2026-03-14 09:44:27.215461 | controller |       | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.215465 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:47 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215469 | controller |       | + '[' 2 == 2 ']'\n    logger.go:42: 09:41:47 | watcher-cinder/1-deploy-watcher-no-cinder
2026-03-14 09:44:27.215474 | controller |       | test step completed 1-deploy-watcher-no-cinder\n    logger.go:42: 09:41:47 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215484 | controller |       | starting test step 2-deploy-cinder\n    logger.go:42: 09:41:47 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215489 | controller |       | OpenStackControlPlane:watcher-kuttl-default/openstack updated\n    logger.go:42:
2026-03-14 09:44:27.215493 | controller |       09:41:47 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:44:27.215498 | controller |       \       # check that the decision detects that there is a cinder service and\n        #
2026-03-14 09:44:27.215502 | controller |       does not log that storage collector is skipped\n        [ \"$(oc logs -n $NAMESPACE
2026-03-14 09:44:27.215506 | controller |       watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled,
2026-03-14 09:44:27.215510 | controller |       skipping storage collector')\" == 0 ]\n        ]\n    logger.go:42: 09:41:47 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215514 | controller |       | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.215518 | controller |       09:41:47 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.215522 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:41:47 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215527 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:41:48 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215531 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215537 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.215542 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.215551 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.215557 | controller |       0 ]\n        ]\n    logger.go:42: 09:41:48 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.215563 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.215569 | controller |       09:41:48 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.215575 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:41:48 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215581 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:41:49 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215588 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215593 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.215597 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.215601 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.215606 | controller |       0 ]\n        ]\n    logger.go:42: 09:41:49 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.215610 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.215614 | controller |       09:41:49 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.215618 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:41:49 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215623 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:41:50 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215627 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215631 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.215635 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.215640 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.215644 | controller |       0 ]\n        ]\n    logger.go:42: 09:41:50 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.215648 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.215652 | controller |       09:41:50 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.215657 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:41:50 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215661 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:41:51 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215665 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215669 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.215673 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.215678 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.215682 | controller |       0 ]\n        ]\n    logger.go:42: 09:41:51 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.215686 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.215690 | controller |       09:41:51 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.215701 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:41:52 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215706 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:41:53 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215710 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215715 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.215719 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.215723 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.215728 | controller |       0 ]\n        ]\n    logger.go:42: 09:41:53 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.215732 | controller |       grep -c 'Block storage service is not enabled, skipping storage collector'\n    logger.go:42:
2026-03-14 09:44:27.215736 | controller |       09:41:53 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.215740 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:41:53 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215749 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:41:54 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215753 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215757 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.215763 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.215767 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.215771 | controller |       0 ]\n        ]\n    logger.go:42: 09:41:54 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.215776 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.215780 | controller |       09:41:54 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.215784 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:41:54 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215788 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:41:55 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215792 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215797 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.215801 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.215805 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.215809 | controller |       0 ]\n        ]\n    logger.go:42: 09:41:55 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.215814 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.215818 | controller |       09:41:55 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.215822 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:41:55 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215826 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:41:56 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215830 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215835 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.215839 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.215843 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.215847 | controller |       0 ]\n        ]\n    logger.go:42: 09:41:56 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.215851 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.215856 | controller |       09:41:56 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.215860 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:41:56 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215864 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:41:57 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215868 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215872 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.215876 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.215881 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.215885 | controller |       0 ]\n        ]\n    logger.go:42: 09:41:57 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.215889 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.215896 | controller |       09:41:57 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.215906 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:41:58 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215913 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:41:59 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215919 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215925 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.215931 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.215938 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.215943 | controller |       0 ]\n        ]\n    logger.go:42: 09:41:59 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.215947 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.215951 | controller |       09:41:59 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.215955 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:41:59 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215960 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:00 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.215965 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.215970 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.215974 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.215979 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.215985 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:00 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.215989 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.215993 | controller |       09:42:00 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.215998 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:00 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216002 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:01 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216006 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216010 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216014 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216019 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216023 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:01 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216027 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.216031 | controller |       09:42:01 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.216036 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:01 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216040 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:02 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216044 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216049 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216053 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216057 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216063 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:02 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216067 | controller |       grep -c 'Block storage service is not enabled, skipping storage collector'\n    logger.go:42:
2026-03-14 09:44:27.216071 | controller |       09:42:02 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.216075 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:42:02 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216079 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:03 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216083 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216088 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216092 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216098 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216102 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:03 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216106 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.216111 | controller |       09:42:03 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.216123 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:03 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216127 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:05 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216132 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216136 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216140 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216144 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216149 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:05 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216153 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.216157 | controller |       09:42:05 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.216161 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:05 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216165 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:06 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216170 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216175 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216181 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216187 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216193 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:06 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216198 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.216204 | controller |       09:42:06 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.216210 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:06 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216216 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:07 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216222 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216228 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216233 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216248 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216254 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:07 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216260 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.216265 | controller |       09:42:07 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.216272 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:07 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216278 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:08 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216284 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216313 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216320 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216326 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216332 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:08 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216338 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.216344 | controller |       09:42:08 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.216350 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:08 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216355 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:09 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216359 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216364 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216368 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216375 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216379 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:09 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216383 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.216388 | controller |       09:42:09 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.216393 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:09 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216406 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:10 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216411 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216415 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216419 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216423 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216428 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:10 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216433 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.216439 | controller |       09:42:10 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.216445 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:11 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216451 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:12 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216456 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216465 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216471 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216477 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216483 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:12 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216489 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.216494 | controller |       09:42:12 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.216500 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:12 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216506 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:13 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216512 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216518 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216524 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216530 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216536 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:13 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216542 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.216547 | controller |       09:42:13 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.216553 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:13 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216559 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:14 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216565 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216571 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216577 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216583 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216589 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:14 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216594 | controller |       grep -c 'Block storage service is not enabled, skipping storage collector'\n    logger.go:42:
2026-03-14 09:44:27.216598 | controller |       09:42:14 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.216603 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:42:14 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216610 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:15 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216615 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216619 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216623 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216627 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216631 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:15 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216635 | controller |       grep -c 'Block storage service is not enabled, skipping storage collector'\n    logger.go:42:
2026-03-14 09:44:27.216640 | controller |       09:42:15 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.216644 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:42:15 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216657 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:16 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216662 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216666 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216671 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216676 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216682 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:16 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216688 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.216694 | controller |       09:42:16 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.216700 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:16 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216706 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:17 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216711 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216717 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216723 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216729 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216735 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:17 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216741 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.216747 | controller |       09:42:17 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.216753 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:18 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216758 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:19 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216764 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216770 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216775 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216781 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216787 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:19 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216791 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.216795 | controller |       09:42:19 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.216799 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:19 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216804 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:20 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216808 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216812 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216816 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216820 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216827 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:20 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216831 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.216837 | controller |       09:42:20 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.216841 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:20 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216845 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:21 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216850 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216854 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216858 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216862 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216866 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:21 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216870 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.216875 | controller |       09:42:21 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.216879 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:21 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216891 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:22 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216895 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216899 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216904 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216908 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216912 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:22 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216916 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.216920 | controller |       09:42:22 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.216924 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:22 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216929 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:23 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216933 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216937 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216942 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216947 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.216953 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:23 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.216959 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.216965 | controller |       09:42:23 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.216971 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:24 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216977 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:25 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.216983 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.216988 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.216993 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.216997 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.217001 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:25 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.217009 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.217014 | controller |       09:42:25 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.217023 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:25 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217033 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:26 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217039 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.217045 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.217051 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.217057 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.217063 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:26 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.217069 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.217075 | controller |       09:42:26 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.217080 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:26 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217086 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:27 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217091 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.217095 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.217099 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.217104 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.217108 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:27 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.217112 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.217117 | controller |       09:42:27 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.217123 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:27 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217129 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:28 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217144 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.217151 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.217157 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.217162 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.217168 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:28 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.217174 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.217179 | controller |       09:42:28 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.217184 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:28 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217188 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:29 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217192 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.217197 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.217203 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.217212 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.217218 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:29 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.217224 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.217230 | controller |       09:42:29 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.217235 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:30 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217241 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:31 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217247 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.217253 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.217259 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.217267 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.217273 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:31 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.217279 | controller |       grep -c 'Block storage service is not enabled, skipping storage collector'\n    logger.go:42:
2026-03-14 09:44:27.217285 | controller |       09:42:31 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.217312 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:42:31 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217318 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:32 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217325 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.217331 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.217337 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.217342 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.217348 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:32 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.217354 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.217360 | controller |       09:42:32 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.217366 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:32 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217371 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:33 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217377 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.217383 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.217389 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.217394 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.217400 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:33 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.217406 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.217412 | controller |       09:42:33 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.217418 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:33 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217422 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:34 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217434 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.217442 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.217447 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.217451 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.217455 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:34 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.217460 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.217464 | controller |       09:42:34 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.217468 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:34 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217472 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:35 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217477 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.217481 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.217485 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.217489 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.217493 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:35 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.217498 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.217502 | controller |       09:42:35 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.217506 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:36 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217516 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:37 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217520 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.217530 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.217534 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.217539 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.217543 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:37 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.217547 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.217551 | controller |       09:42:37 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.217556 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:37 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217560 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:38 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217564 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.217568 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.217573 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.217578 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.217584 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:38 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.217590 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.217596 | controller |       09:42:38 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.217602 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:38 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217611 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:39 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217617 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.217623 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.217628 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.217633 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.217638 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:39 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.217642 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.217646 | controller |       09:42:39 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.217650 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:39 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217655 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:40 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217666 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.217671 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.217677 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.217683 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.217689 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:40 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.217695 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.217701 | controller |       09:42:40 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.217707 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:40 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217713 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:41 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217780 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.217785 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.217793 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.217799 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.217803 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:41 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.217807 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.217812 | controller |       09:42:41 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.217816 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:42 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217820 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:43 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217825 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.217831 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.217837 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.217842 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.217848 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:43 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.217854 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.217860 | controller |       09:42:43 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.217869 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:43 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217875 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:44 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217880 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.217886 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.217892 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.217898 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.217904 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:44 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.217910 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.217915 | controller |       09:42:44 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.217921 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:44 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217927 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:45 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217933 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.217938 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.217944 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.217950 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.217956 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:45 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.217962 | controller |       grep -c 'Block storage service is not enabled, skipping storage collector'\n    logger.go:42:
2026-03-14 09:44:27.217968 | controller |       09:42:45 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.217973 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:42:45 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217978 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:46 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.217990 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.217995 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.217999 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.218003 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.218007 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:46 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.218011 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.218015 | controller |       09:42:46 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.218020 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:46 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218026 | controller |       | + '[' 2 == 0 ']'\n    logger.go:42: 09:42:47 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218031 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.218035 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.218039 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.218043 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.218047 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:47 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.218053 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.218057 | controller |       09:42:47 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.218062 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:48 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218066 | controller |       | Error from server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\"
2026-03-14 09:44:27.218070 | controller |       is waiting to start: ContainerCreating\n    logger.go:42: 09:42:48 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218074 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:42:49 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218078 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.218083 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.218087 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.218091 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.218095 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:49 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.218099 | controller |       grep -c 'Block storage service is not enabled, skipping storage collector'\n    logger.go:42:
2026-03-14 09:44:27.218103 | controller |       09:42:49 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.218107 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:42:49 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218112 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:42:50 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218116 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.218120 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.218124 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.218128 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.218132 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:50 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.218136 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.218141 | controller |       09:42:50 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.218145 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:50 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218149 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:42:51 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218153 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.218157 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.218161 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.218165 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.218170 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:51 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.218174 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.218178 | controller |       09:42:51 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.218188 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:51 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218192 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:42:52 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218196 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.218202 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.218206 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.218211 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.218216 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:52 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.218221 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.218225 | controller |       09:42:52 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.218229 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:52 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218233 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:42:54 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218237 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.218242 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.218246 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.218250 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.218254 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:54 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.218258 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.218263 | controller |       09:42:54 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.218267 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:54 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218271 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:42:55 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218275 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.218279 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.218283 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.218300 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.218307 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:55 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.218312 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.218316 | controller |       09:42:55 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.218321 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:55 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218325 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:42:56 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218329 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.218333 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.218337 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.218341 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.218346 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:56 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.218350 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.218354 | controller |       09:42:56 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.218360 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:56 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218366 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:42:57 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218370 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.218374 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.218379 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.218383 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.218387 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:57 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.218391 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.218395 | controller |       09:42:57 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.218408 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:58 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218413 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:42:59 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218419 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.218425 | controller |       detects that there is a cinder service and\n        # does not log that storage
2026-03-14 09:44:27.218431 | controller |       collector is skipped\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.218437 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.218443 | controller |       0 ]\n        ]\n    logger.go:42: 09:42:59 | watcher-cinder/2-deploy-cinder | ++
2026-03-14 09:44:27.218449 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.218455 | controller |       09:42:59 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.218461 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:42:59 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218465 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:42:59 | watcher-cinder/2-deploy-cinder
2026-03-14 09:44:27.218469 | controller |       | test step completed 2-deploy-cinder\n    logger.go:42: 09:42:59 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218474 | controller |       | starting test step 3-remove-cinder\n    logger.go:42: 09:42:59 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218478 | controller |       | OpenStackControlPlane:watcher-kuttl-default/openstack updated\n    logger.go:42:
2026-03-14 09:44:27.218484 | controller |       09:42:59 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:44:27.218490 | controller |       \       # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.218496 | controller |       service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep
2026-03-14 09:44:27.218502 | controller |       -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n
2026-03-14 09:44:27.218507 | controller |       \       ]\n    logger.go:42: 09:42:59 | watcher-cinder/3-remove-cinder | ++ oc logs
2026-03-14 09:44:27.218513 | controller |       -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42: 09:42:59
2026-03-14 09:44:27.218519 | controller |       | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled,
2026-03-14 09:44:27.218525 | controller |       skipping storage collector'\n    logger.go:42: 09:42:59 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218531 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:43:00 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218535 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.218540 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.218544 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is
2026-03-14 09:44:27.218551 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.218555 | controller |       09:43:00 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.218559 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:43:00 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218564 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.218568 | controller |       \   logger.go:42: 09:43:00 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n
2026-03-14 09:44:27.218572 | controller |       \   logger.go:42: 09:43:01 | watcher-cinder/3-remove-cinder | running command: [sh
2026-03-14 09:44:27.218576 | controller |       -c set -euxo pipefail\n        # check that the decision engine correctly detects
2026-03-14 09:44:27.218581 | controller |       that there is no cinder service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.218585 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.218589 | controller |       2 ]\n        ]\n    logger.go:42: 09:43:01 | watcher-cinder/3-remove-cinder | ++
2026-03-14 09:44:27.218593 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.218598 | controller |       09:43:01 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.218602 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:43:01 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218608 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:43:02 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218613 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.218617 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.218621 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is
2026-03-14 09:44:27.218625 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.218629 | controller |       09:43:02 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.218634 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:43:02 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218645 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.218651 | controller |       \   logger.go:42: 09:43:03 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n
2026-03-14 09:44:27.218655 | controller |       \   logger.go:42: 09:43:04 | watcher-cinder/3-remove-cinder | running command: [sh
2026-03-14 09:44:27.218659 | controller |       -c set -euxo pipefail\n        # check that the decision engine correctly detects
2026-03-14 09:44:27.218663 | controller |       that there is no cinder service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.218667 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.218672 | controller |       2 ]\n        ]\n    logger.go:42: 09:43:04 | watcher-cinder/3-remove-cinder | ++
2026-03-14 09:44:27.218676 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.218680 | controller |       09:43:04 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.218684 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:43:04 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218688 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:43:05 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218692 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.218697 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.218701 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is
2026-03-14 09:44:27.218705 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.218711 | controller |       09:43:05 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.218715 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:43:05 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218719 | controller |       | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.218723 | controller |       09:43:05 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n    logger.go:42:
2026-03-14 09:44:27.218727 | controller |       09:43:06 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:44:27.218732 | controller |       \       # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.218736 | controller |       service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep
2026-03-14 09:44:27.218740 | controller |       -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n
2026-03-14 09:44:27.218744 | controller |       \       ]\n    logger.go:42: 09:43:06 | watcher-cinder/3-remove-cinder | ++ oc logs
2026-03-14 09:44:27.218748 | controller |       -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42: 09:43:06
2026-03-14 09:44:27.218753 | controller |       | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled,
2026-03-14 09:44:27.218757 | controller |       skipping storage collector'\n    logger.go:42: 09:43:06 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218761 | controller |       | Error from server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\"
2026-03-14 09:44:27.218766 | controller |       is waiting to start: ContainerCreating\n    logger.go:42: 09:43:06 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218772 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:43:07 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218780 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.218786 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.218792 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is
2026-03-14 09:44:27.218798 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.218804 | controller |       09:43:07 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.218809 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:43:07 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218814 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.218818 | controller |       \   logger.go:42: 09:43:07 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n
2026-03-14 09:44:27.218822 | controller |       \   logger.go:42: 09:43:08 | watcher-cinder/3-remove-cinder | running command: [sh
2026-03-14 09:44:27.218826 | controller |       -c set -euxo pipefail\n        # check that the decision engine correctly detects
2026-03-14 09:44:27.218830 | controller |       that there is no cinder service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.218835 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.218839 | controller |       2 ]\n        ]\n    logger.go:42: 09:43:08 | watcher-cinder/3-remove-cinder | ++
2026-03-14 09:44:27.218843 | controller |       grep -c 'Block storage service is not enabled, skipping storage collector'\n    logger.go:42:
2026-03-14 09:44:27.218854 | controller |       09:43:08 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.218858 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:43:08 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218863 | controller |       | + '[' 0 == 2 ']'\n    logger.go:42: 09:43:09 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218867 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.218871 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.218877 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is
2026-03-14 09:44:27.218882 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.218886 | controller |       09:43:09 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.218890 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:43:09 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218894 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.218899 | controller |       \   logger.go:42: 09:43:10 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n
2026-03-14 09:44:27.218905 | controller |       \   logger.go:42: 09:43:11 | watcher-cinder/3-remove-cinder | running command: [sh
2026-03-14 09:44:27.218910 | controller |       -c set -euxo pipefail\n        # check that the decision engine correctly detects
2026-03-14 09:44:27.218915 | controller |       that there is no cinder service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.218920 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.218925 | controller |       2 ]\n        ]\n    logger.go:42: 09:43:11 | watcher-cinder/3-remove-cinder | ++
2026-03-14 09:44:27.218929 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.218935 | controller |       09:43:11 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.218940 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:43:11 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218944 | controller |       | + '[' 2 == 2 ']'\n    logger.go:42: 09:43:12 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218948 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.218952 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.218956 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is
2026-03-14 09:44:27.218961 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.218966 | controller |       09:43:12 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.218970 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:43:12 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.218975 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.218979 | controller |       \   logger.go:42: 09:43:12 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n
2026-03-14 09:44:27.218983 | controller |       \   logger.go:42: 09:43:13 | watcher-cinder/3-remove-cinder | running command: [sh
2026-03-14 09:44:27.218987 | controller |       -c set -euxo pipefail\n        # check that the decision engine correctly detects
2026-03-14 09:44:27.218991 | controller |       that there is no cinder service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.218996 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.219000 | controller |       2 ]\n        ]\n    logger.go:42: 09:43:13 | watcher-cinder/3-remove-cinder | ++
2026-03-14 09:44:27.219004 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.219008 | controller |       09:43:13 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.219012 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:43:13 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.219016 | controller |       | + '[' 2 == 2 ']'\n    logger.go:42: 09:43:14 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.219020 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.219025 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.219030 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is
2026-03-14 09:44:27.219035 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.219039 | controller |       09:43:14 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.219043 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:43:14 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.219047 | controller |       | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.219051 | controller |       09:43:14 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n    logger.go:42:
2026-03-14 09:44:27.219056 | controller |       09:43:15 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n
2026-03-14 09:44:27.219065 | controller |       \       # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.219070 | controller |       service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep
2026-03-14 09:44:27.219075 | controller |       -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n
2026-03-14 09:44:27.219079 | controller |       \       ]\n    logger.go:42: 09:43:15 | watcher-cinder/3-remove-cinder | ++ oc logs
2026-03-14 09:44:27.219083 | controller |       -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42: 09:43:15
2026-03-14 09:44:27.219087 | controller |       | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is not enabled,
2026-03-14 09:44:27.219092 | controller |       skipping storage collector'\n    logger.go:42: 09:43:16 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.219096 | controller |       | + '[' 2 == 2 ']'\n    logger.go:42: 09:43:17 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.219100 | controller |       | running command: [sh -c set -euxo pipefail\n        # check that the decision
2026-03-14 09:44:27.219104 | controller |       engine correctly detects that there is no cinder service\n        [ \"$(oc logs
2026-03-14 09:44:27.219108 | controller |       -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is
2026-03-14 09:44:27.219112 | controller |       not enabled, skipping storage collector')\" == 2 ]\n        ]\n    logger.go:42:
2026-03-14 09:44:27.219117 | controller |       09:43:17 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.219121 | controller |       watcher-kuttl-decision-engine-0\n    logger.go:42: 09:43:17 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.219127 | controller |       | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n
2026-03-14 09:44:27.219131 | controller |       \   logger.go:42: 09:43:17 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n
2026-03-14 09:44:27.219136 | controller |       \   logger.go:42: 09:43:18 | watcher-cinder/3-remove-cinder | running command: [sh
2026-03-14 09:44:27.219142 | controller |       -c set -euxo pipefail\n        # check that the decision engine correctly detects
2026-03-14 09:44:27.219147 | controller |       that there is no cinder service\n        [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.219153 | controller |       |grep -c 'Block storage service is not enabled, skipping storage collector')\" ==
2026-03-14 09:44:27.219159 | controller |       2 ]\n        ]\n    logger.go:42: 09:43:18 | watcher-cinder/3-remove-cinder | ++
2026-03-14 09:44:27.219165 | controller |       oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n    logger.go:42:
2026-03-14 09:44:27.219171 | controller |       09:43:18 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is
2026-03-14 09:44:27.219179 | controller |       not enabled, skipping storage collector'\n    logger.go:42: 09:43:18 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.219185 | controller |       | + '[' 2 == 2 ']'\n    logger.go:42: 09:43:18 | watcher-cinder/3-remove-cinder
2026-03-14 09:44:27.219189 | controller |       | test step completed 3-remove-cinder\n    logger.go:42: 09:43:18 | watcher-cinder/4-cleanup-watcher
2026-03-14 09:44:27.219193 | controller |       | starting test step 4-cleanup-watcher\n    logger.go:42: 09:43:30 | watcher-cinder/4-cleanup-watcher
2026-03-14 09:44:27.219199 | controller |       | test step completed 4-cleanup-watcher\n    logger.go:42: 09:43:30 | watcher-cinder
2026-03-14 09:44:27.219203 | controller |       | skipping kubernetes event logging\n=== CONT  kuttl/harness/watcher-api-scaling\n
2026-03-14 09:44:27.219208 | controller |       \   logger.go:42: 09:43:30 | watcher-api-scaling | Skipping creation of user-supplied
2026-03-14 09:44:27.219212 | controller |       namespace: watcher-kuttl-default\n    logger.go:42: 09:43:30 | watcher-api-scaling/0-cleanup-watcher
2026-03-14 09:44:27.219216 | controller |       | starting test step 0-cleanup-watcher\n    logger.go:42: 09:43:30 | watcher-api-scaling/0-cleanup-watcher
2026-03-14 09:44:27.219220 | controller |       | test step completed 0-cleanup-watcher\n    logger.go:42: 09:43:30 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219224 | controller |       | starting test step 1-deploy-with-defaults\n    logger.go:42: 09:43:30 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219228 | controller |       | Watcher:watcher-kuttl-default/watcher-kuttl created\n    logger.go:42: 09:43:30
2026-03-14 09:44:27.219232 | controller |       | watcher-api-scaling/1-deploy-with-defaults | running command: [sh -c set -euxo
2026-03-14 09:44:27.219237 | controller |       pipefail\n        oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:44:27.219242 | controller |       service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n        SERVICEID=$(oc
2026-03-14 09:44:27.219248 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.219254 | controller |       -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n        [ $(oc get -n watcher-kuttl-default
2026-03-14 09:44:27.219260 | controller |       keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n        [
2026-03-14 09:44:27.219265 | controller |       -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\"
2026-03-14 09:44:27.219271 | controller |       ]\n        # If we are running the container locally, skip following test\n        if
2026-03-14 09:44:27.219286 | controller |       [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\"
2026-03-14 09:44:27.219312 | controller |       == \"\" ]; then\n            exit 0\n        fi\n        env_variables=$(oc set
2026-03-14 09:44:27.219316 | controller |       env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)
2026-03-14 09:44:27.219321 | controller |       -n openstack-operators --list)\n        counter=0\n        for i in ${env_variables};
2026-03-14 09:44:27.219325 | controller |       do\n            if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n                echo
2026-03-14 09:44:27.219329 | controller |       ${i}\n                counter=$((counter + 1))\n            fi\n        done\n        if
2026-03-14 09:44:27.219333 | controller |       [ ${counter} -lt 3 ]; then\n          echo \"Error: Less than 3 _URL_DEFAULT variables
2026-03-14 09:44:27.219337 | controller |       found.\"\n          exit 1\n        else\n          echo \"Success: ${counter} _URL_DEFAULT
2026-03-14 09:44:27.219341 | controller |       variables found.\"\n        fi\n        ]\n    logger.go:42: 09:43:30 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219346 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.219350 | controller |       value -c Name -c Type\n    logger.go:42: 09:43:30 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219354 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:43:32 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219358 | controller |       | + '[' 0 == 1 ']'\n    logger.go:42: 09:43:33 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219362 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.219366 | controller |       openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c
2026-03-14 09:44:27.219371 | controller |       ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.219377 | controller |       -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print
2026-03-14 09:44:27.219381 | controller |       $1}')\n        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.219385 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl
2026-03-14 09:44:27.219390 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # If we are running the container
2026-03-14 09:44:27.219394 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:44:27.219398 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:44:27.219405 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:44:27.219411 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:44:27.219416 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:44:27.219422 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:44:27.219427 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:44:27.219433 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:44:27.219439 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:44:27.219445 | controller |       \       ]\n    logger.go:42: 09:43:33 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219451 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.219455 | controller |       value -c Name -c Type\n    logger.go:42: 09:43:33 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219460 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:43:35 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219464 | controller |       | + '[' 0 == 1 ']'\n    logger.go:42: 09:43:37 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219468 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.219472 | controller |       openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c
2026-03-14 09:44:27.219476 | controller |       ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.219480 | controller |       -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print
2026-03-14 09:44:27.219485 | controller |       $1}')\n        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.219489 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl
2026-03-14 09:44:27.219493 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # If we are running the container
2026-03-14 09:44:27.219497 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:44:27.219501 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:44:27.219512 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:44:27.219517 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:44:27.219521 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:44:27.219525 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:44:27.219529 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:44:27.219534 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:44:27.219538 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:44:27.219544 | controller |       \       ]\n    logger.go:42: 09:43:37 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219548 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.219552 | controller |       value -c Name -c Type\n    logger.go:42: 09:43:37 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219557 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:43:39 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219561 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:43:39 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219565 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:43:39 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219569 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.219573 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:43:39 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219578 | controller |       | ++ grep watcher\n    logger.go:42: 09:43:41 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219582 | controller |       | + SERVICEID=86b564fd237a41c2b773bfe0f05b75f1\n    logger.go:42: 09:43:41 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219590 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.219594 | controller |       \   logger.go:42: 09:43:41 | watcher-api-scaling/1-deploy-with-defaults | + '['
2026-03-14 09:44:27.219598 | controller |       86b564fd237a41c2b773bfe0f05b75f1 == 86b564fd237a41c2b773bfe0f05b75f1 ']'\n    logger.go:42:
2026-03-14 09:44:27.219603 | controller |       09:43:41 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.219607 | controller |       watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n    logger.go:42: 09:43:41
2026-03-14 09:44:27.219611 | controller |       | watcher-api-scaling/1-deploy-with-defaults | + '[' -n '' ']'\n    logger.go:42:
2026-03-14 09:44:27.219615 | controller |       09:43:42 | watcher-api-scaling/1-deploy-with-defaults | running command: [sh -c
2026-03-14 09:44:27.219619 | controller |       set -euxo pipefail\n        oc exec -n watcher-kuttl-default openstackclient --
2026-03-14 09:44:27.219623 | controller |       openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n
2026-03-14 09:44:27.219628 | controller |       \       SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:44:27.219632 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n
2026-03-14 09:44:27.219636 | controller |       \       [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.219640 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl
2026-03-14 09:44:27.219644 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # If we are running the container
2026-03-14 09:44:27.219663 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:44:27.219668 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:44:27.219672 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:44:27.219676 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:44:27.219680 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:44:27.219684 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:44:27.219689 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:44:27.219693 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:44:27.219697 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:44:27.219703 | controller |       \       ]\n    logger.go:42: 09:43:42 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219707 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.219711 | controller |       value -c Name -c Type\n    logger.go:42: 09:43:42 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219715 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:43:44 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219726 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:43:44 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219730 | controller |       | ++ grep watcher\n    logger.go:42: 09:43:44 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219734 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:43:44 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219739 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.219745 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:43:46 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219751 | controller |       | + SERVICEID=86b564fd237a41c2b773bfe0f05b75f1\n    logger.go:42: 09:43:46 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219756 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.219762 | controller |       \   logger.go:42: 09:43:47 | watcher-api-scaling/1-deploy-with-defaults | + '['
2026-03-14 09:44:27.219768 | controller |       86b564fd237a41c2b773bfe0f05b75f1 == 86b564fd237a41c2b773bfe0f05b75f1 ']'\n    logger.go:42:
2026-03-14 09:44:27.219774 | controller |       09:43:47 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.219780 | controller |       watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n    logger.go:42: 09:43:47
2026-03-14 09:44:27.219787 | controller |       | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.219793 | controller |       ']'\n    logger.go:42: 09:43:47 | watcher-api-scaling/1-deploy-with-defaults | ++
2026-03-14 09:44:27.219798 | controller |       oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:44:27.219804 | controller |       \   logger.go:42: 09:43:47 | watcher-api-scaling/1-deploy-with-defaults | + '['
2026-03-14 09:44:27.219810 | controller |       '' == '' ']'\n    logger.go:42: 09:43:47 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219816 | controller |       | + exit 0\n    logger.go:42: 09:43:48 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219822 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.219829 | controller |       openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c
2026-03-14 09:44:27.219835 | controller |       ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.219841 | controller |       -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print
2026-03-14 09:44:27.219847 | controller |       $1}')\n        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.219853 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl
2026-03-14 09:44:27.219859 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # If we are running the container
2026-03-14 09:44:27.219865 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:44:27.219871 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:44:27.219878 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:44:27.219884 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:44:27.219893 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:44:27.219899 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:44:27.219905 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:44:27.219913 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:44:27.219920 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:44:27.219927 | controller |       \       ]\n    logger.go:42: 09:43:48 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219935 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.219941 | controller |       value -c Name -c Type\n    logger.go:42: 09:43:48 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219948 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:43:50 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219954 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:43:50 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219960 | controller |       | ++ grep watcher\n    logger.go:42: 09:43:50 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219966 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:43:50 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219976 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.219982 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:43:52 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.219999 | controller |       | + SERVICEID=86b564fd237a41c2b773bfe0f05b75f1\n    logger.go:42: 09:43:52 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220006 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.220013 | controller |       \   logger.go:42: 09:43:52 | watcher-api-scaling/1-deploy-with-defaults | + '['
2026-03-14 09:44:27.220019 | controller |       86b564fd237a41c2b773bfe0f05b75f1 == 86b564fd237a41c2b773bfe0f05b75f1 ']'\n    logger.go:42:
2026-03-14 09:44:27.220026 | controller |       09:43:52 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.220032 | controller |       watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n    logger.go:42: 09:43:52
2026-03-14 09:44:27.220038 | controller |       | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.220044 | controller |       ']'\n    logger.go:42: 09:43:52 | watcher-api-scaling/1-deploy-with-defaults | ++
2026-03-14 09:44:27.220050 | controller |       oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:44:27.220055 | controller |       \   logger.go:42: 09:43:52 | watcher-api-scaling/1-deploy-with-defaults | + '['
2026-03-14 09:44:27.220061 | controller |       '' == '' ']'\n    logger.go:42: 09:43:52 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220066 | controller |       | + exit 0\n    logger.go:42: 09:43:53 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220072 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.220077 | controller |       openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c
2026-03-14 09:44:27.220083 | controller |       ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.220088 | controller |       -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print
2026-03-14 09:44:27.220094 | controller |       $1}')\n        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.220102 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl
2026-03-14 09:44:27.220111 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # If we are running the container
2026-03-14 09:44:27.220117 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:44:27.220123 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:44:27.220128 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:44:27.220134 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:44:27.220139 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:44:27.220144 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:44:27.220151 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:44:27.220157 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:44:27.220162 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:44:27.220168 | controller |       \       ]\n    logger.go:42: 09:43:53 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220174 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.220180 | controller |       value -c Name -c Type\n    logger.go:42: 09:43:53 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220186 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:43:55 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220192 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:43:55 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220198 | controller |       | ++ grep watcher\n    logger.go:42: 09:43:55 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220204 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.220210 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:43:55 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220216 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:43:58 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220221 | controller |       | + SERVICEID=86b564fd237a41c2b773bfe0f05b75f1\n    logger.go:42: 09:43:58 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220227 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.220233 | controller |       \   logger.go:42: 09:43:58 | watcher-api-scaling/1-deploy-with-defaults | + '['
2026-03-14 09:44:27.220243 | controller |       86b564fd237a41c2b773bfe0f05b75f1 == 86b564fd237a41c2b773bfe0f05b75f1 ']'\n    logger.go:42:
2026-03-14 09:44:27.220249 | controller |       09:43:58 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.220265 | controller |       watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n    logger.go:42: 09:43:58
2026-03-14 09:44:27.220273 | controller |       | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.220279 | controller |       ']'\n    logger.go:42: 09:43:58 | watcher-api-scaling/1-deploy-with-defaults | ++
2026-03-14 09:44:27.220285 | controller |       oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:44:27.220314 | controller |       \   logger.go:42: 09:43:58 | watcher-api-scaling/1-deploy-with-defaults | + '['
2026-03-14 09:44:27.220321 | controller |       '' == '' ']'\n    logger.go:42: 09:43:58 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220327 | controller |       | + exit 0\n    logger.go:42: 09:43:59 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220333 | controller |       | running command: [sh -c set -euxo pipefail\n        oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.220339 | controller |       openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c
2026-03-14 09:44:27.220352 | controller |       ^watcher) == 1 ]\n        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.220358 | controller |       -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print
2026-03-14 09:44:27.220364 | controller |       $1}')\n        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.220369 | controller |       == ${SERVICEID} ]\n        [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl
2026-03-14 09:44:27.220374 | controller |       -o jsonpath={.status.hash.dbsync})\" ]\n        # If we are running the container
2026-03-14 09:44:27.220378 | controller |       locally, skip following test\n        if [ \"$(oc get pods -n openstack-operators
2026-03-14 09:44:27.220382 | controller |       -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n            exit
2026-03-14 09:44:27.220386 | controller |       0\n        fi\n        env_variables=$(oc set env $(oc get pods -n openstack-operators
2026-03-14 09:44:27.220390 | controller |       -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n
2026-03-14 09:44:27.220394 | controller |       \       counter=0\n        for i in ${env_variables}; do\n            if echo ${i}
2026-03-14 09:44:27.220399 | controller |       | grep '_URL_DEFAULT' &> /dev/null; then\n                echo ${i}\n                counter=$((counter
2026-03-14 09:44:27.220403 | controller |       + 1))\n            fi\n        done\n        if [ ${counter} -lt 3 ]; then\n          echo
2026-03-14 09:44:27.220407 | controller |       \"Error: Less than 3 _URL_DEFAULT variables found.\"\n          exit 1\n        else\n
2026-03-14 09:44:27.220411 | controller |       \         echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n        fi\n
2026-03-14 09:44:27.220415 | controller |       \       ]\n    logger.go:42: 09:43:59 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220420 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.220424 | controller |       value -c Name -c Type\n    logger.go:42: 09:43:59 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220428 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:44:01 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220432 | controller |       | + '[' 1 == 1 ']'\n    logger.go:42: 09:44:01 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220436 | controller |       | ++ grep watcher\n    logger.go:42: 09:44:01 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220440 | controller |       | ++ awk '{print $1}'\n    logger.go:42: 09:44:01 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220445 | controller |       | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.220449 | controller |       -f value -c Name -c Type -c ID\n    logger.go:42: 09:44:03 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220453 | controller |       | + SERVICEID=86b564fd237a41c2b773bfe0f05b75f1\n    logger.go:42: 09:44:03 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220457 | controller |       | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n
2026-03-14 09:44:27.220461 | controller |       \   logger.go:42: 09:44:04 | watcher-api-scaling/1-deploy-with-defaults | + '['
2026-03-14 09:44:27.220466 | controller |       86b564fd237a41c2b773bfe0f05b75f1 == 86b564fd237a41c2b773bfe0f05b75f1 ']'\n    logger.go:42:
2026-03-14 09:44:27.220470 | controller |       09:44:04 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.220474 | controller |       watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n    logger.go:42: 09:44:04
2026-03-14 09:44:27.220478 | controller |       | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.220482 | controller |       ']'\n    logger.go:42: 09:44:04 | watcher-api-scaling/1-deploy-with-defaults | ++
2026-03-14 09:44:27.220487 | controller |       oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n
2026-03-14 09:44:27.220500 | controller |       \   logger.go:42: 09:44:04 | watcher-api-scaling/1-deploy-with-defaults | + '['
2026-03-14 09:44:27.220505 | controller |       '' == '' ']'\n    logger.go:42: 09:44:04 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220510 | controller |       | + exit 0\n    logger.go:42: 09:44:04 | watcher-api-scaling/1-deploy-with-defaults
2026-03-14 09:44:27.220516 | controller |       | test step completed 1-deploy-with-defaults\n    logger.go:42: 09:44:04 | watcher-api-scaling/2-scale-up-watcher-api
2026-03-14 09:44:27.220522 | controller |       | starting test step 2-scale-up-watcher-api\n    logger.go:42: 09:44:04 | watcher-api-scaling/2-scale-up-watcher-api
2026-03-14 09:44:27.220528 | controller |       | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json'
2026-03-14 09:44:27.220534 | controller |       -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/replicas\", \"value\":3}]'\n
2026-03-14 09:44:27.220540 | controller |       \       ]\n    logger.go:42: 09:44:04 | watcher-api-scaling/2-scale-up-watcher-api
2026-03-14 09:44:27.220546 | controller |       | watcher.watcher.openstack.org/watcher-kuttl patched\n    logger.go:42: 09:44:15
2026-03-14 09:44:27.220551 | controller |       | watcher-api-scaling/2-scale-up-watcher-api | test step completed 2-scale-up-watcher-api\n
2026-03-14 09:44:27.220557 | controller |       \   logger.go:42: 09:44:15 | watcher-api-scaling/3-scale-down-watcher-api | starting
2026-03-14 09:44:27.220563 | controller |       test step 3-scale-down-watcher-api\n    logger.go:42: 09:44:15 | watcher-api-scaling/3-scale-down-watcher-api
2026-03-14 09:44:27.220568 | controller |       | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json'
2026-03-14 09:44:27.220573 | controller |       -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/replicas\", \"value\":1}]'\n
2026-03-14 09:44:27.220577 | controller |       \       ]\n    logger.go:42: 09:44:16 | watcher-api-scaling/3-scale-down-watcher-api
2026-03-14 09:44:27.220581 | controller |       | watcher.watcher.openstack.org/watcher-kuttl patched\n    logger.go:42: 09:44:19
2026-03-14 09:44:27.220585 | controller |       | watcher-api-scaling/3-scale-down-watcher-api | test step completed 3-scale-down-watcher-api\n
2026-03-14 09:44:27.220589 | controller |       \   logger.go:42: 09:44:19 | watcher-api-scaling/4-scale-down-zero-watcher-api |
2026-03-14 09:44:27.220593 | controller |       starting test step 4-scale-down-zero-watcher-api\n    logger.go:42: 09:44:19 | watcher-api-scaling/4-scale-down-zero-watcher-api
2026-03-14 09:44:27.220598 | controller |       | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json'
2026-03-14 09:44:27.220605 | controller |       -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/replicas\", \"value\":0}]'\n
2026-03-14 09:44:27.220609 | controller |       \       ]\n    logger.go:42: 09:44:19 | watcher-api-scaling/4-scale-down-zero-watcher-api
2026-03-14 09:44:27.220614 | controller |       | watcher.watcher.openstack.org/watcher-kuttl patched\n    logger.go:42: 09:44:20
2026-03-14 09:44:27.220618 | controller |       | watcher-api-scaling/4-scale-down-zero-watcher-api | test step completed 4-scale-down-zero-watcher-api\n
2026-03-14 09:44:27.220623 | controller |       \   logger.go:42: 09:44:20 | watcher-api-scaling/5-cleanup-watcher | starting test
2026-03-14 09:44:27.220628 | controller |       step 5-cleanup-watcher\n    logger.go:42: 09:44:20 | watcher-api-scaling/5-cleanup-watcher
2026-03-14 09:44:27.220632 | controller |       | running command: [sh -c set -ex\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.220637 | controller |       -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0
2026-03-14 09:44:27.220643 | controller |       ]\n        ]\n    logger.go:42: 09:44:20 | watcher-api-scaling/5-cleanup-watcher
2026-03-14 09:44:27.220649 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.220655 | controller |       value -c Name -c Type\n    logger.go:42: 09:44:20 | watcher-api-scaling/5-cleanup-watcher
2026-03-14 09:44:27.220660 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:44:22 | watcher-api-scaling/5-cleanup-watcher
2026-03-14 09:44:27.220666 | controller |       | + '[' 1 == 0 ']'\n    logger.go:42: 09:44:24 | watcher-api-scaling/5-cleanup-watcher
2026-03-14 09:44:27.220675 | controller |       | running command: [sh -c set -ex\n        oc exec -n watcher-kuttl-default openstackclient
2026-03-14 09:44:27.220681 | controller |       -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0
2026-03-14 09:44:27.220687 | controller |       ]\n        ]\n    logger.go:42: 09:44:24 | watcher-api-scaling/5-cleanup-watcher
2026-03-14 09:44:27.220693 | controller |       | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f
2026-03-14 09:44:27.220699 | controller |       value -c Name -c Type\n    logger.go:42: 09:44:24 | watcher-api-scaling/5-cleanup-watcher
2026-03-14 09:44:27.220705 | controller |       | ++ grep -c '^watcher'\n    logger.go:42: 09:44:26 | watcher-api-scaling/5-cleanup-watcher
2026-03-14 09:44:27.220711 | controller |       | + '[' 0 == 0 ']'\n    logger.go:42: 09:44:26 | watcher-api-scaling/5-cleanup-watcher
2026-03-14 09:44:27.220717 | controller |       | test step completed 5-cleanup-watcher\n    logger.go:42: 09:44:26 | watcher-api-scaling
2026-03-14 09:44:27.220722 | controller |       | skipping kubernetes event logging\n=== CONT  kuttl/harness/deps\n    logger.go:42:
2026-03-14 09:44:27.220728 | controller |       09:44:26 | deps | Ignoring infra.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n
2026-03-14 09:44:27.220743 | controller |       \   logger.go:42: 09:44:26 | deps | Ignoring keystone.yaml as it does not match
2026-03-14 09:44:27.220749 | controller |       file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n    logger.go:42: 09:44:26 |
2026-03-14 09:44:27.220755 | controller |       deps | Ignoring kustomization.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n
2026-03-14 09:44:27.220759 | controller |       \   logger.go:42: 09:44:26 | deps | Ignoring namespace.yaml as it does not match
2026-03-14 09:44:27.220763 | controller |       file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n    logger.go:42: 09:44:26 |
2026-03-14 09:44:27.220767 | controller |       deps | Ignoring telemetry.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n
2026-03-14 09:44:27.220771 | controller |       \   logger.go:42: 09:44:26 | deps | Skipping creation of user-supplied namespace:
2026-03-14 09:44:27.220775 | controller |       watcher-kuttl-default\n    logger.go:42: 09:44:26 | deps | skipping kubernetes event
2026-03-14 09:44:27.220780 | controller |       logging\n=== CONT  kuttl/harness/common\n    logger.go:42: 09:44:26 | common | Ignoring
2026-03-14 09:44:27.220784 | controller |       cleanup-assert.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n
2026-03-14 09:44:27.220788 | controller |       \   logger.go:42: 09:44:26 | common | Ignoring cleanup-errors.yaml as it does not
2026-03-14 09:44:27.220792 | controller |       match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n    logger.go:42: 09:44:26
2026-03-14 09:44:27.220796 | controller |       | common | Ignoring cleanup-watcher.yaml as it does not match file name regexp:
2026-03-14 09:44:27.220800 | controller |       ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n    logger.go:42: 09:44:26 | common | Ignoring
2026-03-14 09:44:27.220804 | controller |       deploy-with-defaults.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n
2026-03-14 09:44:27.220809 | controller |       \   logger.go:42: 09:44:26 | common | Skipping creation of user-supplied namespace:
2026-03-14 09:44:27.220813 | controller |       watcher-kuttl-default\n    logger.go:42: 09:44:26 | common | skipping kubernetes
2026-03-14 09:44:27.220817 | controller |       event logging\n=== NAME  kuttl\n    harness.go:406: run tests finished\n    harness.go:514:
2026-03-14 09:44:27.220821 | controller |       cleaning up\n    harness.go:571: removing temp folder: \"\"\n--- FAIL: kuttl (1507.06s)\n
2026-03-14 09:44:27.220825 | controller |       \   --- FAIL: kuttl/harness (0.00s)\n        --- FAIL: kuttl/harness/appcred-tests
2026-03-14 09:44:27.220829 | controller |       (951.33s)\n        --- PASS: kuttl/harness/watcher-notification (40.72s)\n        ---
2026-03-14 09:44:27.220833 | controller |       PASS: kuttl/harness/watcher-topology (33.46s)\n        --- PASS: kuttl/harness/watcher-tls-certs-change
2026-03-14 09:44:27.220837 | controller |       (35.80s)\n        --- PASS: kuttl/harness/watcher-tls (134.34s)\n        --- PASS:
2026-03-14 09:44:27.220844 | controller |       kuttl/harness/watcher-rmquser (34.76s)\n        --- PASS: kuttl/harness/watcher
2026-03-14 09:44:27.220848 | controller |       (88.99s)\n        --- PASS: kuttl/harness/watcher-cinder (131.87s)\n        ---
2026-03-14 09:44:27.220854 | controller |       PASS: kuttl/harness/watcher-api-scaling (55.74s)\n        --- PASS: kuttl/harness/deps
2026-03-14 09:44:27.220859 | controller |       (0.00s)\n        --- PASS: kuttl/harness/common (0.02s)\nFAIL"
2026-03-14 09:44:27.220866 | controller |     stdout_lines:
2026-03-14 09:44:27.220872 | controller |     - oc kuttl test --v 1 --start-kind=false --config test/kuttl/test-suites/default/config.yaml
2026-03-14 09:44:27.220878 | controller |     - === RUN   kuttl
2026-03-14 09:44:27.220884 | controller |     - '    harness.go:463: starting setup'
2026-03-14 09:44:27.220890 | controller |     - '    harness.go:255: running tests using configured kubeconfig.'
2026-03-14 09:44:27.220896 | controller |     - '    harness.go:278: Successful connection to cluster at: https://api.crc.testing:6443'
2026-03-14 09:44:27.220902 | controller |     - '    harness.go:363: running tests'
2026-03-14 09:44:27.220908 | controller |     - '    harness.go:75: going to run test suite with timeout of 300 seconds for each
2026-03-14 09:44:27.220914 | controller |       step'
2026-03-14 09:44:27.220925 | controller |     - '    harness.go:375: testsuite: test/kuttl/test-suites/default/ has 11 tests'
2026-03-14 09:44:27.220932 | controller |     - === RUN   kuttl/harness
2026-03-14 09:44:27.220938 | controller |     - === RUN   kuttl/harness/appcred-tests
2026-03-14 09:44:27.220943 | controller |     - === PAUSE kuttl/harness/appcred-tests
2026-03-14 09:44:27.220950 | controller |     - === RUN   kuttl/harness/common
2026-03-14 09:44:27.220957 | controller |     - === PAUSE kuttl/harness/common
2026-03-14 09:44:27.220964 | controller |     - === RUN   kuttl/harness/deps
2026-03-14 09:44:27.220970 | controller |     - === PAUSE kuttl/harness/deps
2026-03-14 09:44:27.220978 | controller |     - === RUN   kuttl/harness/watcher
2026-03-14 09:44:27.220984 | controller |     - === PAUSE kuttl/harness/watcher
2026-03-14 09:44:27.220990 | controller |     - === RUN   kuttl/harness/watcher-api-scaling
2026-03-14 09:44:27.220996 | controller |     - === PAUSE kuttl/harness/watcher-api-scaling
2026-03-14 09:44:27.221002 | controller |     - === RUN   kuttl/harness/watcher-cinder
2026-03-14 09:44:27.221008 | controller |     - === PAUSE kuttl/harness/watcher-cinder
2026-03-14 09:44:27.221014 | controller |     - === RUN   kuttl/harness/watcher-notification
2026-03-14 09:44:27.221020 | controller |     - === PAUSE kuttl/harness/watcher-notification
2026-03-14 09:44:27.221026 | controller |     - === RUN   kuttl/harness/watcher-rmquser
2026-03-14 09:44:27.221031 | controller |     - === PAUSE kuttl/harness/watcher-rmquser
2026-03-14 09:44:27.221037 | controller |     - === RUN   kuttl/harness/watcher-tls
2026-03-14 09:44:27.221043 | controller |     - === PAUSE kuttl/harness/watcher-tls
2026-03-14 09:44:27.221049 | controller |     - === RUN   kuttl/harness/watcher-tls-certs-change
2026-03-14 09:44:27.221055 | controller |     - === PAUSE kuttl/harness/watcher-tls-certs-change
2026-03-14 09:44:27.221060 | controller |     - === RUN   kuttl/harness/watcher-topology
2026-03-14 09:44:27.221066 | controller |     - === PAUSE kuttl/harness/watcher-topology
2026-03-14 09:44:27.221080 | controller |     - === CONT  kuttl/harness/appcred-tests
2026-03-14 09:44:27.221087 | controller |     - '    logger.go:42: 09:19:19 | appcred-tests | Skipping creation of user-supplied
2026-03-14 09:44:27.221093 | controller |       namespace: watcher-kuttl-default'
2026-03-14 09:44:27.221100 | controller |     - '    logger.go:42: 09:19:19 | appcred-tests/0-cleanup | starting test step 0-cleanup'
2026-03-14 09:44:27.221104 | controller |     - '    logger.go:42: 09:19:19 | appcred-tests/0-cleanup | test step completed 0-cleanup'
2026-03-14 09:44:27.221108 | controller |     - '    logger.go:42: 09:19:19 | appcred-tests/1-deploy | starting test step 1-deploy'
2026-03-14 09:44:27.221113 | controller |     - '    logger.go:42: 09:19:19 | appcred-tests/1-deploy | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:44:27.221117 | controller |       created'
2026-03-14 09:44:27.221124 | controller |     - '    logger.go:42: 09:19:19 | appcred-tests/1-deploy | running command: [sh -c set
2026-03-14 09:44:27.221128 | controller |       -euo pipefail'
2026-03-14 09:44:27.221133 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.221140 | controller |     - '        oc wait -n "${NS}" watcher watcher-kuttl --for=condition=Ready --timeout=600s]'
2026-03-14 09:44:27.221145 | controller |     - '    logger.go:42: 09:20:09 | appcred-tests/1-deploy | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:44:27.221149 | controller |       condition met'
2026-03-14 09:44:27.221153 | controller |     - '    logger.go:42: 09:20:09 | appcred-tests/1-deploy | test step completed 1-deploy'
2026-03-14 09:44:27.221158 | controller |     - '    logger.go:42: 09:20:09 | appcred-tests/2-deploy-appcred | starting test step
2026-03-14 09:44:27.221162 | controller |       2-deploy-appcred'
2026-03-14 09:44:27.221166 | controller |     - '    logger.go:42: 09:20:09 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.221171 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.221175 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.221179 | controller |     - '        '
2026-03-14 09:44:27.221183 | controller |     - '        old_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.221188 | controller |     - '        old_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.221192 | controller |     - '        old_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.221196 | controller |     - '        oc create configmap appcred-watcher-pre \'
2026-03-14 09:44:27.221200 | controller |     - '          --from-literal=api_uid="${old_api_uid}" \'
2026-03-14 09:44:27.221205 | controller |     - '          --from-literal=applier_uid="${old_applier_uid}" \'
2026-03-14 09:44:27.221209 | controller |     - '          --from-literal=decision_engine_uid="${old_de_uid}" \'
2026-03-14 09:44:27.221213 | controller |     - '          --dry-run=client -o yaml | oc apply -n "${NS}" -f -'
2026-03-14 09:44:27.221217 | controller |     - '        '
2026-03-14 09:44:27.221223 | controller |     - '        cat <<''EOF'' | oc apply -n "${NS}" -f -'
2026-03-14 09:44:27.221227 | controller |     - '        apiVersion: keystone.openstack.org/v1beta1'
2026-03-14 09:44:27.221232 | controller |     - '        kind: KeystoneApplicationCredential'
2026-03-14 09:44:27.221236 | controller |     - '        metadata:'
2026-03-14 09:44:27.221240 | controller |     - '          name: ac-watcher'
2026-03-14 09:44:27.221244 | controller |     - '        spec:'
2026-03-14 09:44:27.221249 | controller |     - '          secret: osp-secret'
2026-03-14 09:44:27.221253 | controller |     - '          passwordSelector: WatcherPassword'
2026-03-14 09:44:27.221257 | controller |     - '          userName: watcher'
2026-03-14 09:44:27.221261 | controller |     - '          roles:'
2026-03-14 09:44:27.221266 | controller |     - '            - admin'
2026-03-14 09:44:27.221270 | controller |     - '            - service'
2026-03-14 09:44:27.221275 | controller |     - '          unrestricted: false'
2026-03-14 09:44:27.221279 | controller |     - '        EOF'
2026-03-14 09:44:27.221283 | controller |     - '        '
2026-03-14 09:44:27.221290 | controller |     - '        oc patch watcher watcher-kuttl -n "${NS}" --type=merge -p ''{"spec":{"auth":{"applicationCredentialSecret":"ac-watcher-secret"}}}'']'
2026-03-14 09:44:27.221312 | controller |     - '    logger.go:42: 09:20:10 | appcred-tests/2-deploy-appcred | configmap/appcred-watcher-pre
2026-03-14 09:44:27.221318 | controller |       created'
2026-03-14 09:44:27.221324 | controller |     - '    logger.go:42: 09:20:10 | appcred-tests/2-deploy-appcred | keystoneapplicationcredential.keystone.openstack.org/ac-watcher
2026-03-14 09:44:27.221330 | controller |       created'
2026-03-14 09:44:27.221336 | controller |     - '    logger.go:42: 09:20:10 | appcred-tests/2-deploy-appcred | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:44:27.221342 | controller |       patched'
2026-03-14 09:44:27.221348 | controller |     - '    logger.go:42: 09:20:10 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.221355 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.221359 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.221363 | controller |     - '        '
2026-03-14 09:44:27.221367 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.221372 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.221376 | controller |     - '        '
2026-03-14 09:44:27.221382 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.221386 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.221391 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.221395 | controller |     - '          exit 1'
2026-03-14 09:44:27.221399 | controller |     - '        fi'
2026-03-14 09:44:27.221403 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.221407 | controller |     - '        '
2026-03-14 09:44:27.221412 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.221416 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.221420 | controller |       | base64 -d)'
2026-03-14 09:44:27.221424 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.221428 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.221432 | controller |     - '          exit 1'
2026-03-14 09:44:27.221444 | controller |     - '        fi'
2026-03-14 09:44:27.221449 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.221453 | controller |     - '        '
2026-03-14 09:44:27.221457 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.221461 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.221466 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.221470 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.221474 | controller |       --timeout=300s'
2026-03-14 09:44:27.221478 | controller |     - '        '
2026-03-14 09:44:27.221482 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.221486 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.221491 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.221495 | controller |     - '        '
2026-03-14 09:44:27.221499 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.221503 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.221507 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.221512 | controller |     - '        '
2026-03-14 09:44:27.221516 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.221520 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.221524 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.221529 | controller |       available"'
2026-03-14 09:44:27.221533 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.221537 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.221541 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.221546 | controller |     - '          exit 1'
2026-03-14 09:44:27.221550 | controller |     - '        fi'
2026-03-14 09:44:27.221554 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.221558 | controller |     - '        '
2026-03-14 09:44:27.221563 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.221567 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.221571 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.221577 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.221581 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.221586 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.221590 | controller |       -- \'
2026-03-14 09:44:27.221594 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.221598 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.221603 | controller |     - '    logger.go:42: 09:20:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.221607 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.221611 | controller |     - '    logger.go:42: 09:25:10 | appcred-tests/2-deploy-appcred | error: timed out
2026-03-14 09:44:27.221616 | controller |       waiting for the condition on keystoneapplicationcredentials/ac-watcher'
2026-03-14 09:44:27.221620 | controller |     - '    logger.go:42: 09:25:11 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.221624 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.221628 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.221633 | controller |     - '        '
2026-03-14 09:44:27.221637 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.221641 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.221648 | controller |     - '        '
2026-03-14 09:44:27.221653 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.221657 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.221662 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.221666 | controller |     - '          exit 1'
2026-03-14 09:44:27.221670 | controller |     - '        fi'
2026-03-14 09:44:27.221674 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.221679 | controller |     - '        '
2026-03-14 09:44:27.221683 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.221687 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.221691 | controller |       | base64 -d)'
2026-03-14 09:44:27.221696 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.221701 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.221706 | controller |     - '          exit 1'
2026-03-14 09:44:27.221710 | controller |     - '        fi'
2026-03-14 09:44:27.221714 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.221725 | controller |     - '        '
2026-03-14 09:44:27.221746 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.221752 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.221757 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.221763 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.221769 | controller |       --timeout=300s'
2026-03-14 09:44:27.221775 | controller |     - '        '
2026-03-14 09:44:27.221781 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.221787 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.221793 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.221797 | controller |     - '        '
2026-03-14 09:44:27.221802 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.221808 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.221813 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.221817 | controller |     - '        '
2026-03-14 09:44:27.221823 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.221828 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.221832 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.221836 | controller |       available"'
2026-03-14 09:44:27.221841 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.221845 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.221849 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.221854 | controller |     - '          exit 1'
2026-03-14 09:44:27.221859 | controller |     - '        fi'
2026-03-14 09:44:27.221863 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.221867 | controller |     - '        '
2026-03-14 09:44:27.221871 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.221876 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.221880 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.221884 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.221888 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.221893 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.221897 | controller |       -- \'
2026-03-14 09:44:27.221901 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.221906 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.221913 | controller |     - '    logger.go:42: 09:25:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.221918 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.221924 | controller |     - '    logger.go:42: 09:25:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.221931 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.221938 | controller |       found'
2026-03-14 09:44:27.221944 | controller |     - '    logger.go:42: 09:25:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.221950 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.221957 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.221963 | controller |     - '        '
2026-03-14 09:44:27.221969 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.221975 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.221981 | controller |     - '        '
2026-03-14 09:44:27.221989 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.221996 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.222002 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.222008 | controller |     - '          exit 1'
2026-03-14 09:44:27.222014 | controller |     - '        fi'
2026-03-14 09:44:27.222021 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.222027 | controller |     - '        '
2026-03-14 09:44:27.222034 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.222040 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.222050 | controller |       | base64 -d)'
2026-03-14 09:44:27.222057 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.222063 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.222069 | controller |     - '          exit 1'
2026-03-14 09:44:27.222075 | controller |     - '        fi'
2026-03-14 09:44:27.222081 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.222087 | controller |     - '        '
2026-03-14 09:44:27.222102 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.222107 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.222111 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.222115 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.222119 | controller |       --timeout=300s'
2026-03-14 09:44:27.222123 | controller |     - '        '
2026-03-14 09:44:27.222128 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.222132 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.222136 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.222140 | controller |     - '        '
2026-03-14 09:44:27.222144 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.222149 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.222153 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.222157 | controller |     - '        '
2026-03-14 09:44:27.222161 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.222165 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.222170 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.222174 | controller |       available"'
2026-03-14 09:44:27.222178 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.222182 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.222186 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.222191 | controller |     - '          exit 1'
2026-03-14 09:44:27.222197 | controller |     - '        fi'
2026-03-14 09:44:27.222202 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.222208 | controller |     - '        '
2026-03-14 09:44:27.222214 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.222220 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.222226 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.222232 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.222237 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.222243 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.222249 | controller |       -- \'
2026-03-14 09:44:27.222255 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.222261 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.222270 | controller |     - '    logger.go:42: 09:25:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.222276 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.222282 | controller |     - '    logger.go:42: 09:25:12 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.222303 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.222313 | controller |       found'
2026-03-14 09:44:27.222317 | controller |     - '    logger.go:42: 09:25:13 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.222321 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.222326 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.222330 | controller |     - '        '
2026-03-14 09:44:27.222334 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.222338 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.222342 | controller |     - '        '
2026-03-14 09:44:27.222347 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.222351 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.222355 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.222359 | controller |     - '          exit 1'
2026-03-14 09:44:27.222366 | controller |     - '        fi'
2026-03-14 09:44:27.222370 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.222374 | controller |     - '        '
2026-03-14 09:44:27.222381 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.222385 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.222389 | controller |       | base64 -d)'
2026-03-14 09:44:27.222393 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.222398 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.222402 | controller |     - '          exit 1'
2026-03-14 09:44:27.222406 | controller |     - '        fi'
2026-03-14 09:44:27.222410 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.222414 | controller |     - '        '
2026-03-14 09:44:27.222419 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.222431 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.222436 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.222440 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.222444 | controller |       --timeout=300s'
2026-03-14 09:44:27.222448 | controller |     - '        '
2026-03-14 09:44:27.222453 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.222457 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.222461 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.222465 | controller |     - '        '
2026-03-14 09:44:27.222469 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.222474 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.222478 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.222482 | controller |     - '        '
2026-03-14 09:44:27.222486 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.222490 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.222496 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.222502 | controller |       available"'
2026-03-14 09:44:27.222506 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.222511 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.222515 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.222519 | controller |     - '          exit 1'
2026-03-14 09:44:27.222523 | controller |     - '        fi'
2026-03-14 09:44:27.222527 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.222532 | controller |     - '        '
2026-03-14 09:44:27.222536 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.222540 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.222544 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.222548 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.222553 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.222557 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.222561 | controller |       -- \'
2026-03-14 09:44:27.222565 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.222569 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.222573 | controller |     - '    logger.go:42: 09:25:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.222578 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.222582 | controller |     - '    logger.go:42: 09:25:14 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.222586 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.222590 | controller |       found'
2026-03-14 09:44:27.222594 | controller |     - '    logger.go:42: 09:25:15 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.222598 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.222603 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.222607 | controller |     - '        '
2026-03-14 09:44:27.222611 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.222615 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.222619 | controller |     - '        '
2026-03-14 09:44:27.222624 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.222628 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.222632 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.222636 | controller |     - '          exit 1'
2026-03-14 09:44:27.222640 | controller |     - '        fi'
2026-03-14 09:44:27.222645 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.222649 | controller |     - '        '
2026-03-14 09:44:27.222653 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.222657 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.222661 | controller |       | base64 -d)'
2026-03-14 09:44:27.222666 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.222670 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.222674 | controller |     - '          exit 1'
2026-03-14 09:44:27.222678 | controller |     - '        fi'
2026-03-14 09:44:27.222682 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.222687 | controller |     - '        '
2026-03-14 09:44:27.222694 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.222704 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.222708 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.222712 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.222717 | controller |       --timeout=300s'
2026-03-14 09:44:27.222721 | controller |     - '        '
2026-03-14 09:44:27.222725 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.222729 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.222733 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.222738 | controller |     - '        '
2026-03-14 09:44:27.222742 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.222746 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.222750 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.222754 | controller |     - '        '
2026-03-14 09:44:27.222759 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.222763 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.222767 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.222771 | controller |       available"'
2026-03-14 09:44:27.222775 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.222780 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.222784 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.222788 | controller |     - '          exit 1'
2026-03-14 09:44:27.222792 | controller |     - '        fi'
2026-03-14 09:44:27.222796 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.222800 | controller |     - '        '
2026-03-14 09:44:27.222805 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.222809 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.222813 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.222817 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.222822 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.222826 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.222830 | controller |       -- \'
2026-03-14 09:44:27.222834 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.222838 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.222843 | controller |     - '    logger.go:42: 09:25:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.222848 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.222852 | controller |     - '    logger.go:42: 09:25:15 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.222856 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.222860 | controller |       found'
2026-03-14 09:44:27.222864 | controller |     - '    logger.go:42: 09:25:16 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.222870 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.222874 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.222879 | controller |     - '        '
2026-03-14 09:44:27.222883 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.222887 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.222891 | controller |     - '        '
2026-03-14 09:44:27.222895 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.222899 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.222904 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.222908 | controller |     - '          exit 1'
2026-03-14 09:44:27.222912 | controller |     - '        fi'
2026-03-14 09:44:27.222918 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.222923 | controller |     - '        '
2026-03-14 09:44:27.222927 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.222932 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.222937 | controller |       | base64 -d)'
2026-03-14 09:44:27.222942 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.222946 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.222950 | controller |     - '          exit 1'
2026-03-14 09:44:27.222955 | controller |     - '        fi'
2026-03-14 09:44:27.222961 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.222965 | controller |     - '        '
2026-03-14 09:44:27.222969 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.222973 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.222982 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.222988 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.222992 | controller |       --timeout=300s'
2026-03-14 09:44:27.222996 | controller |     - '        '
2026-03-14 09:44:27.223000 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.223005 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.223009 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.223013 | controller |     - '        '
2026-03-14 09:44:27.223017 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.223022 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.223026 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.223030 | controller |     - '        '
2026-03-14 09:44:27.223034 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.223040 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.223044 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.223049 | controller |       available"'
2026-03-14 09:44:27.223053 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.223057 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.223062 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.223066 | controller |     - '          exit 1'
2026-03-14 09:44:27.223070 | controller |     - '        fi'
2026-03-14 09:44:27.223074 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.223082 | controller |     - '        '
2026-03-14 09:44:27.223086 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.223091 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.223095 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.223099 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.223103 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.223108 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.223112 | controller |       -- \'
2026-03-14 09:44:27.223116 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.223121 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.223126 | controller |     - '    logger.go:42: 09:25:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.223130 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.223135 | controller |     - '    logger.go:42: 09:25:16 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.223139 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.223143 | controller |       found'
2026-03-14 09:44:27.223147 | controller |     - '    logger.go:42: 09:25:17 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.223152 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.223156 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.223160 | controller |     - '        '
2026-03-14 09:44:27.223164 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.223169 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.223173 | controller |     - '        '
2026-03-14 09:44:27.223177 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.223181 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.223186 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.223190 | controller |     - '          exit 1'
2026-03-14 09:44:27.223194 | controller |     - '        fi'
2026-03-14 09:44:27.223198 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.223202 | controller |     - '        '
2026-03-14 09:44:27.223207 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.223211 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.223215 | controller |       | base64 -d)'
2026-03-14 09:44:27.223219 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.223223 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.223228 | controller |     - '          exit 1'
2026-03-14 09:44:27.223232 | controller |     - '        fi'
2026-03-14 09:44:27.223236 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.223240 | controller |     - '        '
2026-03-14 09:44:27.223244 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.223249 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.223253 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.223262 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.223267 | controller |       --timeout=300s'
2026-03-14 09:44:27.223271 | controller |     - '        '
2026-03-14 09:44:27.223276 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.223284 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.223301 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.223308 | controller |     - '        '
2026-03-14 09:44:27.223313 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.223317 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.223321 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.223325 | controller |     - '        '
2026-03-14 09:44:27.223329 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.223334 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.223338 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.223342 | controller |       available"'
2026-03-14 09:44:27.223346 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.223350 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.223355 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.223359 | controller |     - '          exit 1'
2026-03-14 09:44:27.223363 | controller |     - '        fi'
2026-03-14 09:44:27.223367 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.223371 | controller |     - '        '
2026-03-14 09:44:27.223376 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.223380 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.223384 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.223388 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.223392 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.223396 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.223401 | controller |       -- \'
2026-03-14 09:44:27.223405 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.223409 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.223413 | controller |     - '    logger.go:42: 09:25:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.223418 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.223422 | controller |     - '    logger.go:42: 09:25:17 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.223428 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.223433 | controller |       found'
2026-03-14 09:44:27.223440 | controller |     - '    logger.go:42: 09:25:18 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.223446 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.223451 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.223457 | controller |     - '        '
2026-03-14 09:44:27.223463 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.223470 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.223476 | controller |     - '        '
2026-03-14 09:44:27.223481 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.223490 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.223496 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.223502 | controller |     - '          exit 1'
2026-03-14 09:44:27.223507 | controller |     - '        fi'
2026-03-14 09:44:27.223513 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.223519 | controller |     - '        '
2026-03-14 09:44:27.223524 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.223530 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.223535 | controller |       | base64 -d)'
2026-03-14 09:44:27.223540 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.223546 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.223550 | controller |     - '          exit 1'
2026-03-14 09:44:27.223554 | controller |     - '        fi'
2026-03-14 09:44:27.223559 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.223563 | controller |     - '        '
2026-03-14 09:44:27.223567 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.223571 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.223576 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.223588 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.223593 | controller |       --timeout=300s'
2026-03-14 09:44:27.223597 | controller |     - '        '
2026-03-14 09:44:27.223601 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.223606 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.223610 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.223614 | controller |     - '        '
2026-03-14 09:44:27.223618 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.223622 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.223627 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.223631 | controller |     - '        '
2026-03-14 09:44:27.223635 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.223639 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.223643 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.223648 | controller |       available"'
2026-03-14 09:44:27.223652 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.223659 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.223663 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.223667 | controller |     - '          exit 1'
2026-03-14 09:44:27.223671 | controller |     - '        fi'
2026-03-14 09:44:27.223676 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.223680 | controller |     - '        '
2026-03-14 09:44:27.223685 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.223689 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.223694 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.223698 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.223702 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.223708 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.223712 | controller |       -- \'
2026-03-14 09:44:27.223716 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.223721 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.223725 | controller |     - '    logger.go:42: 09:25:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.223729 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.223733 | controller |     - '    logger.go:42: 09:25:18 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.223737 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.223741 | controller |       found'
2026-03-14 09:44:27.223746 | controller |     - '    logger.go:42: 09:25:19 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.223750 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.223754 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.223758 | controller |     - '        '
2026-03-14 09:44:27.223762 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.223767 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.223771 | controller |     - '        '
2026-03-14 09:44:27.223775 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.223779 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.223783 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.223787 | controller |     - '          exit 1'
2026-03-14 09:44:27.223792 | controller |     - '        fi'
2026-03-14 09:44:27.223796 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.223800 | controller |     - '        '
2026-03-14 09:44:27.223804 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.223808 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.223812 | controller |       | base64 -d)'
2026-03-14 09:44:27.223817 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.223821 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.223825 | controller |     - '          exit 1'
2026-03-14 09:44:27.223829 | controller |     - '        fi'
2026-03-14 09:44:27.223834 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.223838 | controller |     - '        '
2026-03-14 09:44:27.223842 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.223846 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.223851 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.223856 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.223861 | controller |       --timeout=300s'
2026-03-14 09:44:27.223890 | controller |     - '        '
2026-03-14 09:44:27.223903 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.223911 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.223917 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.223923 | controller |     - '        '
2026-03-14 09:44:27.223929 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.223935 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.223946 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.223952 | controller |     - '        '
2026-03-14 09:44:27.223958 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.223962 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.223967 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.223971 | controller |       available"'
2026-03-14 09:44:27.223975 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.223979 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.223984 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.223988 | controller |     - '          exit 1'
2026-03-14 09:44:27.223992 | controller |     - '        fi'
2026-03-14 09:44:27.223996 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.224001 | controller |     - '        '
2026-03-14 09:44:27.224005 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.224009 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.224013 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.224017 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.224021 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.224026 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.224030 | controller |       -- \'
2026-03-14 09:44:27.224034 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.224038 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.224043 | controller |     - '    logger.go:42: 09:25:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.224047 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.224051 | controller |     - '    logger.go:42: 09:25:19 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.224055 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.224059 | controller |       found'
2026-03-14 09:44:27.224064 | controller |     - '    logger.go:42: 09:25:20 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.224068 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.224072 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.224076 | controller |     - '        '
2026-03-14 09:44:27.224080 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.224085 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.224089 | controller |     - '        '
2026-03-14 09:44:27.224093 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.224097 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.224101 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.224106 | controller |     - '          exit 1'
2026-03-14 09:44:27.224114 | controller |     - '        fi'
2026-03-14 09:44:27.224118 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.224123 | controller |     - '        '
2026-03-14 09:44:27.224127 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.224131 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.224137 | controller |       | base64 -d)'
2026-03-14 09:44:27.224141 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.224145 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.224150 | controller |     - '          exit 1'
2026-03-14 09:44:27.224154 | controller |     - '        fi'
2026-03-14 09:44:27.224158 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.224165 | controller |     - '        '
2026-03-14 09:44:27.224170 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.224174 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.224178 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.224182 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.224186 | controller |       --timeout=300s'
2026-03-14 09:44:27.224191 | controller |     - '        '
2026-03-14 09:44:27.224203 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.224207 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.224213 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.224217 | controller |     - '        '
2026-03-14 09:44:27.224222 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.224226 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.224230 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.224234 | controller |     - '        '
2026-03-14 09:44:27.224238 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.224243 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.224247 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.224253 | controller |       available"'
2026-03-14 09:44:27.224259 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.224265 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.224271 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.224277 | controller |     - '          exit 1'
2026-03-14 09:44:27.224283 | controller |     - '        fi'
2026-03-14 09:44:27.224305 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.224314 | controller |     - '        '
2026-03-14 09:44:27.224318 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.224322 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.224326 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.224333 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.224338 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.224342 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.224346 | controller |       -- \'
2026-03-14 09:44:27.224350 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.224355 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.224359 | controller |     - '    logger.go:42: 09:25:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.224365 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.224369 | controller |     - '    logger.go:42: 09:25:20 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.224373 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.224378 | controller |       found'
2026-03-14 09:44:27.224382 | controller |     - '    logger.go:42: 09:25:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.224386 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.224390 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.224395 | controller |     - '        '
2026-03-14 09:44:27.224399 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.224403 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.224407 | controller |     - '        '
2026-03-14 09:44:27.224411 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.224416 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.224422 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.224426 | controller |     - '          exit 1'
2026-03-14 09:44:27.224430 | controller |     - '        fi'
2026-03-14 09:44:27.224434 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.224439 | controller |     - '        '
2026-03-14 09:44:27.224443 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.224447 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.224451 | controller |       | base64 -d)'
2026-03-14 09:44:27.224456 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.224460 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.224464 | controller |     - '          exit 1'
2026-03-14 09:44:27.224468 | controller |     - '        fi'
2026-03-14 09:44:27.224473 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.224477 | controller |     - '        '
2026-03-14 09:44:27.224481 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.224485 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.224489 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.224494 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.224498 | controller |       --timeout=300s'
2026-03-14 09:44:27.224502 | controller |     - '        '
2026-03-14 09:44:27.224514 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.224521 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.224525 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.224529 | controller |     - '        '
2026-03-14 09:44:27.224534 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.224538 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.224543 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.224548 | controller |     - '        '
2026-03-14 09:44:27.224554 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.224560 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.224566 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.224574 | controller |       available"'
2026-03-14 09:44:27.224581 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.224587 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.224592 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.224598 | controller |     - '          exit 1'
2026-03-14 09:44:27.224604 | controller |     - '        fi'
2026-03-14 09:44:27.224609 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.224615 | controller |     - '        '
2026-03-14 09:44:27.224621 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.224627 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.224633 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.224639 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.224645 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.224651 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.224656 | controller |       -- \'
2026-03-14 09:44:27.224663 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.224668 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.224674 | controller |     - '    logger.go:42: 09:25:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.224680 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.224686 | controller |     - '    logger.go:42: 09:25:22 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.224691 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.224697 | controller |       found'
2026-03-14 09:44:27.224703 | controller |     - '    logger.go:42: 09:25:23 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.224709 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.224715 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.224721 | controller |     - '        '
2026-03-14 09:44:27.224726 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.224732 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.224738 | controller |     - '        '
2026-03-14 09:44:27.224743 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.224749 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.224755 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.224761 | controller |     - '          exit 1'
2026-03-14 09:44:27.224767 | controller |     - '        fi'
2026-03-14 09:44:27.224773 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.224779 | controller |     - '        '
2026-03-14 09:44:27.224785 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.224791 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.224796 | controller |       | base64 -d)'
2026-03-14 09:44:27.224802 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.224807 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.224811 | controller |     - '          exit 1'
2026-03-14 09:44:27.224819 | controller |     - '        fi'
2026-03-14 09:44:27.224825 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.224831 | controller |     - '        '
2026-03-14 09:44:27.224837 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.224846 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.224852 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.224858 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.224864 | controller |       --timeout=300s'
2026-03-14 09:44:27.224870 | controller |     - '        '
2026-03-14 09:44:27.224877 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.224892 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.224898 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.224904 | controller |     - '        '
2026-03-14 09:44:27.224910 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.224916 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.224922 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.224928 | controller |     - '        '
2026-03-14 09:44:27.224934 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.224939 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.224945 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.224951 | controller |       available"'
2026-03-14 09:44:27.224957 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.224963 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.224969 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.224974 | controller |     - '          exit 1'
2026-03-14 09:44:27.224981 | controller |     - '        fi'
2026-03-14 09:44:27.224989 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.224994 | controller |     - '        '
2026-03-14 09:44:27.225000 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.225005 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.225010 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.225014 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.225018 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.225022 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.225027 | controller |       -- \'
2026-03-14 09:44:27.225031 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.225038 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.225042 | controller |     - '    logger.go:42: 09:25:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.225046 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.225051 | controller |     - '    logger.go:42: 09:25:23 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.225055 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.225059 | controller |       found'
2026-03-14 09:44:27.225063 | controller |     - '    logger.go:42: 09:25:24 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.225068 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.225074 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.225078 | controller |     - '        '
2026-03-14 09:44:27.225082 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.225086 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.225090 | controller |     - '        '
2026-03-14 09:44:27.225095 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.225099 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.225103 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.225107 | controller |     - '          exit 1'
2026-03-14 09:44:27.225112 | controller |     - '        fi'
2026-03-14 09:44:27.225116 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.225120 | controller |     - '        '
2026-03-14 09:44:27.225126 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.225130 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.225134 | controller |       | base64 -d)'
2026-03-14 09:44:27.225138 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.225143 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.225147 | controller |     - '          exit 1'
2026-03-14 09:44:27.225151 | controller |     - '        fi'
2026-03-14 09:44:27.225155 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.225160 | controller |     - '        '
2026-03-14 09:44:27.225164 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.225168 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.225172 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.225176 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.225181 | controller |       --timeout=300s'
2026-03-14 09:44:27.225185 | controller |     - '        '
2026-03-14 09:44:27.225189 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.225202 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.225207 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.225211 | controller |     - '        '
2026-03-14 09:44:27.225215 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.225219 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.225224 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.225228 | controller |     - '        '
2026-03-14 09:44:27.225232 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.225236 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.225241 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.225245 | controller |       available"'
2026-03-14 09:44:27.225250 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.225256 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.225262 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.225268 | controller |     - '          exit 1'
2026-03-14 09:44:27.225276 | controller |     - '        fi'
2026-03-14 09:44:27.225281 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.225303 | controller |     - '        '
2026-03-14 09:44:27.225316 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.225323 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.225329 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.225335 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.225343 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.225347 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.225351 | controller |       -- \'
2026-03-14 09:44:27.225355 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.225360 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.225364 | controller |     - '    logger.go:42: 09:25:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.225368 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.225372 | controller |     - '    logger.go:42: 09:25:24 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.225377 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.225381 | controller |       found'
2026-03-14 09:44:27.225385 | controller |     - '    logger.go:42: 09:25:25 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.225389 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.225393 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.225398 | controller |     - '        '
2026-03-14 09:44:27.225402 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.225406 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.225410 | controller |     - '        '
2026-03-14 09:44:27.225415 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.225419 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.225423 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.225427 | controller |     - '          exit 1'
2026-03-14 09:44:27.225431 | controller |     - '        fi'
2026-03-14 09:44:27.225436 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.225440 | controller |     - '        '
2026-03-14 09:44:27.225444 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.225448 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.225452 | controller |       | base64 -d)'
2026-03-14 09:44:27.225457 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.225461 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.225465 | controller |     - '          exit 1'
2026-03-14 09:44:27.225469 | controller |     - '        fi'
2026-03-14 09:44:27.225473 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.225478 | controller |     - '        '
2026-03-14 09:44:27.225482 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.225486 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.225490 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.225494 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.225499 | controller |       --timeout=300s'
2026-03-14 09:44:27.225503 | controller |     - '        '
2026-03-14 09:44:27.225507 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.225513 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.225526 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.225530 | controller |     - '        '
2026-03-14 09:44:27.225535 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.225539 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.225543 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.225547 | controller |     - '        '
2026-03-14 09:44:27.225552 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.225556 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.225560 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.225564 | controller |       available"'
2026-03-14 09:44:27.225569 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.225573 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.225577 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.225581 | controller |     - '          exit 1'
2026-03-14 09:44:27.225585 | controller |     - '        fi'
2026-03-14 09:44:27.225590 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.225594 | controller |     - '        '
2026-03-14 09:44:27.225598 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.225602 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.225607 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.225611 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.225615 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.225619 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.225623 | controller |       -- \'
2026-03-14 09:44:27.225628 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.225632 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.225636 | controller |     - '    logger.go:42: 09:25:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.225640 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.225648 | controller |     - '    logger.go:42: 09:25:25 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.225652 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.225656 | controller |       found'
2026-03-14 09:44:27.225660 | controller |     - '    logger.go:42: 09:25:26 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.225665 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.225669 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.225673 | controller |     - '        '
2026-03-14 09:44:27.225677 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.225681 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.225686 | controller |     - '        '
2026-03-14 09:44:27.225690 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.225694 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.225700 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.225704 | controller |     - '          exit 1'
2026-03-14 09:44:27.225708 | controller |     - '        fi'
2026-03-14 09:44:27.225713 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.225717 | controller |     - '        '
2026-03-14 09:44:27.225721 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.225725 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.225729 | controller |       | base64 -d)'
2026-03-14 09:44:27.225734 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.225755 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.225760 | controller |     - '          exit 1'
2026-03-14 09:44:27.225764 | controller |     - '        fi'
2026-03-14 09:44:27.225768 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.225772 | controller |     - '        '
2026-03-14 09:44:27.225777 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.225781 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.225785 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.225789 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.225793 | controller |       --timeout=300s'
2026-03-14 09:44:27.225798 | controller |     - '        '
2026-03-14 09:44:27.225802 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.225806 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.225816 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.225821 | controller |     - '        '
2026-03-14 09:44:27.225825 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.225829 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.225833 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.225837 | controller |     - '        '
2026-03-14 09:44:27.225842 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.225846 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.225850 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.225854 | controller |       available"'
2026-03-14 09:44:27.225859 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.225863 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.225867 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.225871 | controller |     - '          exit 1'
2026-03-14 09:44:27.225875 | controller |     - '        fi'
2026-03-14 09:44:27.225879 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.225884 | controller |     - '        '
2026-03-14 09:44:27.225888 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.225892 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.225896 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.225900 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.225905 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.225911 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.225915 | controller |       -- \'
2026-03-14 09:44:27.225919 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.225923 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.225928 | controller |     - '    logger.go:42: 09:25:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.225932 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.225936 | controller |     - '    logger.go:42: 09:25:26 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.225940 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.225944 | controller |       found'
2026-03-14 09:44:27.225949 | controller |     - '    logger.go:42: 09:25:27 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.225953 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.225957 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.225961 | controller |     - '        '
2026-03-14 09:44:27.225966 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.225970 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.225974 | controller |     - '        '
2026-03-14 09:44:27.225978 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.225982 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.225987 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.225991 | controller |     - '          exit 1'
2026-03-14 09:44:27.225995 | controller |     - '        fi'
2026-03-14 09:44:27.225999 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.226004 | controller |     - '        '
2026-03-14 09:44:27.226008 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.226012 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.226016 | controller |       | base64 -d)'
2026-03-14 09:44:27.226020 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.226027 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.226031 | controller |     - '          exit 1'
2026-03-14 09:44:27.226036 | controller |     - '        fi'
2026-03-14 09:44:27.226040 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.226044 | controller |     - '        '
2026-03-14 09:44:27.226048 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.226053 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.226057 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.226061 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.226065 | controller |       --timeout=300s'
2026-03-14 09:44:27.226069 | controller |     - '        '
2026-03-14 09:44:27.226074 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.226078 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.226082 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.226092 | controller |     - '        '
2026-03-14 09:44:27.226096 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.226101 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.226106 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.226111 | controller |     - '        '
2026-03-14 09:44:27.226115 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.226119 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.226123 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.226128 | controller |       available"'
2026-03-14 09:44:27.226132 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.226136 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.226140 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.226144 | controller |     - '          exit 1'
2026-03-14 09:44:27.226149 | controller |     - '        fi'
2026-03-14 09:44:27.226153 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.226157 | controller |     - '        '
2026-03-14 09:44:27.226161 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.226166 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.226170 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.226174 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.226178 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.226182 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.226187 | controller |       -- \'
2026-03-14 09:44:27.226191 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.226195 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.226199 | controller |     - '    logger.go:42: 09:25:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.226204 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.226208 | controller |     - '    logger.go:42: 09:25:27 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.226214 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.226218 | controller |       found'
2026-03-14 09:44:27.226222 | controller |     - '    logger.go:42: 09:25:28 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.226226 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.226231 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.226235 | controller |     - '        '
2026-03-14 09:44:27.226239 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.226243 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.226248 | controller |     - '        '
2026-03-14 09:44:27.226252 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.226256 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.226260 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.226264 | controller |     - '          exit 1'
2026-03-14 09:44:27.226268 | controller |     - '        fi'
2026-03-14 09:44:27.226273 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.226277 | controller |     - '        '
2026-03-14 09:44:27.226281 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.226285 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.226305 | controller |       | base64 -d)'
2026-03-14 09:44:27.226314 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.226320 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.226326 | controller |     - '          exit 1'
2026-03-14 09:44:27.226332 | controller |     - '        fi'
2026-03-14 09:44:27.226338 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.226344 | controller |     - '        '
2026-03-14 09:44:27.226350 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.226356 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.226362 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.226368 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.226374 | controller |       --timeout=300s'
2026-03-14 09:44:27.226384 | controller |     - '        '
2026-03-14 09:44:27.226390 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.226396 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.226402 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.226408 | controller |     - '        '
2026-03-14 09:44:27.226424 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.226432 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.226438 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.226444 | controller |     - '        '
2026-03-14 09:44:27.226450 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.226456 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.226462 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.226469 | controller |       available"'
2026-03-14 09:44:27.226475 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.226484 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.226490 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.226496 | controller |     - '          exit 1'
2026-03-14 09:44:27.226502 | controller |     - '        fi'
2026-03-14 09:44:27.226506 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.226511 | controller |     - '        '
2026-03-14 09:44:27.226515 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.226519 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.226523 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.226528 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.226532 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.226536 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.226541 | controller |       -- \'
2026-03-14 09:44:27.226547 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.226552 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.226558 | controller |     - '    logger.go:42: 09:25:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.226567 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.226573 | controller |     - '    logger.go:42: 09:25:28 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.226579 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.226585 | controller |       found'
2026-03-14 09:44:27.226591 | controller |     - '    logger.go:42: 09:25:29 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.226597 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.226603 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.226609 | controller |     - '        '
2026-03-14 09:44:27.226614 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.226618 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.226623 | controller |     - '        '
2026-03-14 09:44:27.226627 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.226631 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.226637 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.226643 | controller |     - '          exit 1'
2026-03-14 09:44:27.226649 | controller |     - '        fi'
2026-03-14 09:44:27.226655 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.226661 | controller |     - '        '
2026-03-14 09:44:27.226667 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.226673 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.226679 | controller |       | base64 -d)'
2026-03-14 09:44:27.226685 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.226690 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.226696 | controller |     - '          exit 1'
2026-03-14 09:44:27.226701 | controller |     - '        fi'
2026-03-14 09:44:27.226706 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.226710 | controller |     - '        '
2026-03-14 09:44:27.226714 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.226718 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.226722 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.226727 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.226731 | controller |       --timeout=300s'
2026-03-14 09:44:27.226735 | controller |     - '        '
2026-03-14 09:44:27.226739 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.226743 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.226748 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.226752 | controller |     - '        '
2026-03-14 09:44:27.226765 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.226770 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.226774 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.226779 | controller |     - '        '
2026-03-14 09:44:27.226783 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.226791 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.226795 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.226800 | controller |       available"'
2026-03-14 09:44:27.226804 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.226810 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.226814 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.226818 | controller |     - '          exit 1'
2026-03-14 09:44:27.226822 | controller |     - '        fi'
2026-03-14 09:44:27.226827 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.226832 | controller |     - '        '
2026-03-14 09:44:27.226838 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.226844 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.226850 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.226856 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.226861 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.226867 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.226871 | controller |       -- \'
2026-03-14 09:44:27.226875 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.226880 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.226884 | controller |     - '    logger.go:42: 09:25:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.226888 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.226892 | controller |     - '    logger.go:42: 09:25:29 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.226897 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.226901 | controller |       found'
2026-03-14 09:44:27.226907 | controller |     - '    logger.go:42: 09:25:30 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.226912 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.226916 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.226920 | controller |     - '        '
2026-03-14 09:44:27.226924 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.226929 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.226933 | controller |     - '        '
2026-03-14 09:44:27.226937 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.226942 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.226946 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.226950 | controller |     - '          exit 1'
2026-03-14 09:44:27.226954 | controller |     - '        fi'
2026-03-14 09:44:27.226959 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.226963 | controller |     - '        '
2026-03-14 09:44:27.226967 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.226971 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.226976 | controller |       | base64 -d)'
2026-03-14 09:44:27.226982 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.226988 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.226994 | controller |     - '          exit 1'
2026-03-14 09:44:27.227000 | controller |     - '        fi'
2026-03-14 09:44:27.227006 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.227011 | controller |     - '        '
2026-03-14 09:44:27.227018 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.227023 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.227034 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.227040 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.227046 | controller |       --timeout=300s'
2026-03-14 09:44:27.227052 | controller |     - '        '
2026-03-14 09:44:27.227058 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.227064 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.227070 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.227077 | controller |     - '        '
2026-03-14 09:44:27.227085 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.227100 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.227107 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.227112 | controller |     - '        '
2026-03-14 09:44:27.227116 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.227120 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.227124 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.227129 | controller |       available"'
2026-03-14 09:44:27.227133 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.227137 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.227141 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.227146 | controller |     - '          exit 1'
2026-03-14 09:44:27.227150 | controller |     - '        fi'
2026-03-14 09:44:27.227154 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.227158 | controller |     - '        '
2026-03-14 09:44:27.227162 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.227167 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.227171 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.227175 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.227179 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.227183 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.227187 | controller |       -- \'
2026-03-14 09:44:27.227192 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.227196 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.227200 | controller |     - '    logger.go:42: 09:25:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.227204 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.227208 | controller |     - '    logger.go:42: 09:25:31 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.227213 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.227217 | controller |       found'
2026-03-14 09:44:27.227221 | controller |     - '    logger.go:42: 09:25:32 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.227225 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.227229 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.227236 | controller |     - '        '
2026-03-14 09:44:27.227240 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.227245 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.227249 | controller |     - '        '
2026-03-14 09:44:27.227253 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.227257 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.227261 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.227266 | controller |     - '          exit 1'
2026-03-14 09:44:27.227270 | controller |     - '        fi'
2026-03-14 09:44:27.227274 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.227278 | controller |     - '        '
2026-03-14 09:44:27.227282 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.227290 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.227311 | controller |       | base64 -d)'
2026-03-14 09:44:27.227316 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.227320 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.227324 | controller |     - '          exit 1'
2026-03-14 09:44:27.227328 | controller |     - '        fi'
2026-03-14 09:44:27.227332 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.227337 | controller |     - '        '
2026-03-14 09:44:27.227341 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.227345 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.227349 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.227353 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.227357 | controller |       --timeout=300s'
2026-03-14 09:44:27.227362 | controller |     - '        '
2026-03-14 09:44:27.227366 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.227370 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.227374 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.227378 | controller |     - '        '
2026-03-14 09:44:27.227383 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.227393 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.227398 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.227402 | controller |     - '        '
2026-03-14 09:44:27.227407 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.227411 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.227415 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.227419 | controller |       available"'
2026-03-14 09:44:27.227424 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.227428 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.227432 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.227436 | controller |     - '          exit 1'
2026-03-14 09:44:27.227440 | controller |     - '        fi'
2026-03-14 09:44:27.227445 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.227449 | controller |     - '        '
2026-03-14 09:44:27.227453 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.227459 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.227463 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.227468 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.227472 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.227476 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.227483 | controller |       -- \'
2026-03-14 09:44:27.227487 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.227491 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.227497 | controller |     - '    logger.go:42: 09:25:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.227501 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.227505 | controller |     - '    logger.go:42: 09:25:32 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.227509 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.227514 | controller |       found'
2026-03-14 09:44:27.227518 | controller |     - '    logger.go:42: 09:25:33 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.227522 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.227526 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.227530 | controller |     - '        '
2026-03-14 09:44:27.227536 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.227540 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.227545 | controller |     - '        '
2026-03-14 09:44:27.227549 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.227553 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.227557 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.227561 | controller |     - '          exit 1'
2026-03-14 09:44:27.227566 | controller |     - '        fi'
2026-03-14 09:44:27.227570 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.227574 | controller |     - '        '
2026-03-14 09:44:27.227580 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.227584 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.227588 | controller |       | base64 -d)'
2026-03-14 09:44:27.227592 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.227597 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.227601 | controller |     - '          exit 1'
2026-03-14 09:44:27.227605 | controller |     - '        fi'
2026-03-14 09:44:27.227610 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.227614 | controller |     - '        '
2026-03-14 09:44:27.227618 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.227622 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.227627 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.227631 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.227635 | controller |       --timeout=300s'
2026-03-14 09:44:27.227640 | controller |     - '        '
2026-03-14 09:44:27.227644 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.227648 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.227654 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.227658 | controller |     - '        '
2026-03-14 09:44:27.227663 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.227667 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.227678 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.227685 | controller |     - '        '
2026-03-14 09:44:27.227692 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.227698 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.227703 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.227709 | controller |       available"'
2026-03-14 09:44:27.227714 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.227720 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.227726 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.227732 | controller |     - '          exit 1'
2026-03-14 09:44:27.227738 | controller |     - '        fi'
2026-03-14 09:44:27.227744 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.227750 | controller |     - '        '
2026-03-14 09:44:27.227756 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.227762 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.227768 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.227774 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.227780 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.227786 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.227792 | controller |       -- \'
2026-03-14 09:44:27.227798 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.227804 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.227811 | controller |     - '    logger.go:42: 09:25:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.227817 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.227823 | controller |     - '    logger.go:42: 09:25:33 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.227829 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.227836 | controller |       found'
2026-03-14 09:44:27.227842 | controller |     - '    logger.go:42: 09:25:34 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.227848 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.227854 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.227860 | controller |     - '        '
2026-03-14 09:44:27.227866 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.227872 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.227878 | controller |     - '        '
2026-03-14 09:44:27.227884 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.227890 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.227896 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.227904 | controller |     - '          exit 1'
2026-03-14 09:44:27.227909 | controller |     - '        fi'
2026-03-14 09:44:27.227913 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.227917 | controller |     - '        '
2026-03-14 09:44:27.227926 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.227932 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.227937 | controller |       | base64 -d)'
2026-03-14 09:44:27.227943 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.227949 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.227955 | controller |     - '          exit 1'
2026-03-14 09:44:27.227961 | controller |     - '        fi'
2026-03-14 09:44:27.227967 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.227973 | controller |     - '        '
2026-03-14 09:44:27.227978 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.227984 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.227990 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.227996 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.228002 | controller |       --timeout=300s'
2026-03-14 09:44:27.228008 | controller |     - '        '
2026-03-14 09:44:27.228014 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.228020 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.228026 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.228032 | controller |     - '        '
2026-03-14 09:44:27.228038 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.228044 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.228059 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.228065 | controller |     - '        '
2026-03-14 09:44:27.228069 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.228074 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.228078 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.228083 | controller |       available"'
2026-03-14 09:44:27.228087 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.228091 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.228096 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.228100 | controller |     - '          exit 1'
2026-03-14 09:44:27.228104 | controller |     - '        fi'
2026-03-14 09:44:27.228108 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.228112 | controller |     - '        '
2026-03-14 09:44:27.228116 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.228120 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.228125 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.228129 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.228133 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.228137 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.228144 | controller |       -- \'
2026-03-14 09:44:27.228148 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.228152 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.228157 | controller |     - '    logger.go:42: 09:25:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.228161 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.228165 | controller |     - '    logger.go:42: 09:25:34 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.228169 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.228174 | controller |       found'
2026-03-14 09:44:27.228178 | controller |     - '    logger.go:42: 09:25:35 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.228182 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.228186 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.228190 | controller |     - '        '
2026-03-14 09:44:27.228195 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.228201 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.228206 | controller |     - '        '
2026-03-14 09:44:27.228210 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.228214 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.228218 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.228222 | controller |     - '          exit 1'
2026-03-14 09:44:27.228227 | controller |     - '        fi'
2026-03-14 09:44:27.228231 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.228235 | controller |     - '        '
2026-03-14 09:44:27.228239 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.228243 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.228247 | controller |       | base64 -d)'
2026-03-14 09:44:27.228252 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.228256 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.228260 | controller |     - '          exit 1'
2026-03-14 09:44:27.228264 | controller |     - '        fi'
2026-03-14 09:44:27.228268 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.228273 | controller |     - '        '
2026-03-14 09:44:27.228277 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.228281 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.228285 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.228308 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.228313 | controller |       --timeout=300s'
2026-03-14 09:44:27.228317 | controller |     - '        '
2026-03-14 09:44:27.228321 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.228325 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.228329 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.228334 | controller |     - '        '
2026-03-14 09:44:27.228338 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.228342 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.228346 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.228352 | controller |     - '        '
2026-03-14 09:44:27.228363 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.228367 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.228371 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.228376 | controller |       available"'
2026-03-14 09:44:27.228380 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.228384 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.228388 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.228392 | controller |     - '          exit 1'
2026-03-14 09:44:27.228397 | controller |     - '        fi'
2026-03-14 09:44:27.228401 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.228405 | controller |     - '        '
2026-03-14 09:44:27.228409 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.228413 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.228418 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.228422 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.228426 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.228430 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.228435 | controller |       -- \'
2026-03-14 09:44:27.228439 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.228443 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.228447 | controller |     - '    logger.go:42: 09:25:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.228451 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.228456 | controller |     - '    logger.go:42: 09:25:35 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.228460 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.228464 | controller |       found'
2026-03-14 09:44:27.228468 | controller |     - '    logger.go:42: 09:25:36 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.228472 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.228480 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.228484 | controller |     - '        '
2026-03-14 09:44:27.228488 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.228493 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.228497 | controller |     - '        '
2026-03-14 09:44:27.228502 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.228506 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.228510 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.228515 | controller |     - '          exit 1'
2026-03-14 09:44:27.228519 | controller |     - '        fi'
2026-03-14 09:44:27.228523 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.228527 | controller |     - '        '
2026-03-14 09:44:27.228532 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.228536 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.228540 | controller |       | base64 -d)'
2026-03-14 09:44:27.228544 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.228550 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.228554 | controller |     - '          exit 1'
2026-03-14 09:44:27.228559 | controller |     - '        fi'
2026-03-14 09:44:27.228563 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.228567 | controller |     - '        '
2026-03-14 09:44:27.228571 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.228575 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.228580 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.228585 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.228591 | controller |       --timeout=300s'
2026-03-14 09:44:27.228597 | controller |     - '        '
2026-03-14 09:44:27.228603 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.228608 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.228614 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.228620 | controller |     - '        '
2026-03-14 09:44:27.228624 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.228628 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.228636 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.228640 | controller |     - '        '
2026-03-14 09:44:27.228652 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.228657 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.228661 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.228665 | controller |       available"'
2026-03-14 09:44:27.228669 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.228674 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.228678 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.228682 | controller |     - '          exit 1'
2026-03-14 09:44:27.228686 | controller |     - '        fi'
2026-03-14 09:44:27.228691 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.228695 | controller |     - '        '
2026-03-14 09:44:27.228701 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.228705 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.228709 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.228714 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.228718 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.228722 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.228726 | controller |       -- \'
2026-03-14 09:44:27.228731 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.228735 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.228739 | controller |     - '    logger.go:42: 09:25:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.228743 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.228749 | controller |     - '    logger.go:42: 09:25:36 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.228754 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.228758 | controller |       found'
2026-03-14 09:44:27.228762 | controller |     - '    logger.go:42: 09:25:37 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.228766 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.228771 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.228775 | controller |     - '        '
2026-03-14 09:44:27.228779 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.228783 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.228788 | controller |     - '        '
2026-03-14 09:44:27.228793 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.228797 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.228802 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.228806 | controller |     - '          exit 1'
2026-03-14 09:44:27.228810 | controller |     - '        fi'
2026-03-14 09:44:27.228814 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.228818 | controller |     - '        '
2026-03-14 09:44:27.228823 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.228827 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.228831 | controller |       | base64 -d)'
2026-03-14 09:44:27.228835 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.228840 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.228844 | controller |     - '          exit 1'
2026-03-14 09:44:27.228848 | controller |     - '        fi'
2026-03-14 09:44:27.228852 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.228856 | controller |     - '        '
2026-03-14 09:44:27.228861 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.228865 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.228869 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.228873 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.228877 | controller |       --timeout=300s'
2026-03-14 09:44:27.228882 | controller |     - '        '
2026-03-14 09:44:27.228886 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.228890 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.228894 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.228898 | controller |     - '        '
2026-03-14 09:44:27.228904 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.228908 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.228912 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.228917 | controller |     - '        '
2026-03-14 09:44:27.228921 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.228930 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.228936 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.228940 | controller |       available"'
2026-03-14 09:44:27.228944 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.228950 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.228955 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.228959 | controller |     - '          exit 1'
2026-03-14 09:44:27.228963 | controller |     - '        fi'
2026-03-14 09:44:27.228967 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.228971 | controller |     - '        '
2026-03-14 09:44:27.228976 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.228980 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.228984 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.228988 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.228993 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.228997 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.229001 | controller |       -- \'
2026-03-14 09:44:27.229005 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.229009 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.229014 | controller |     - '    logger.go:42: 09:25:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.229018 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.229022 | controller |     - '    logger.go:42: 09:25:38 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.229026 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.229031 | controller |       found'
2026-03-14 09:44:27.229035 | controller |     - '    logger.go:42: 09:25:39 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.229039 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.229043 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.229048 | controller |     - '        '
2026-03-14 09:44:27.229052 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.229056 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.229060 | controller |     - '        '
2026-03-14 09:44:27.229064 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.229069 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.229075 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.229080 | controller |     - '          exit 1'
2026-03-14 09:44:27.229086 | controller |     - '        fi'
2026-03-14 09:44:27.229092 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.229098 | controller |     - '        '
2026-03-14 09:44:27.229104 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.229111 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.229116 | controller |       | base64 -d)'
2026-03-14 09:44:27.229123 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.229128 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.229134 | controller |     - '          exit 1'
2026-03-14 09:44:27.229140 | controller |     - '        fi'
2026-03-14 09:44:27.229146 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.229152 | controller |     - '        '
2026-03-14 09:44:27.229157 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.229163 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.229169 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.229178 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.229184 | controller |       --timeout=300s'
2026-03-14 09:44:27.229190 | controller |     - '        '
2026-03-14 09:44:27.229196 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.229202 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.229207 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.229213 | controller |     - '        '
2026-03-14 09:44:27.229219 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.229224 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.229230 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.229236 | controller |     - '        '
2026-03-14 09:44:27.229243 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.229249 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.229265 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.229272 | controller |       available"'
2026-03-14 09:44:27.229278 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.229301 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.229311 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.229317 | controller |     - '          exit 1'
2026-03-14 09:44:27.229322 | controller |     - '        fi'
2026-03-14 09:44:27.229327 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.229331 | controller |     - '        '
2026-03-14 09:44:27.229335 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.229339 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.229343 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.229348 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.229352 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.229356 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.229360 | controller |       -- \'
2026-03-14 09:44:27.229364 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.229369 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.229373 | controller |     - '    logger.go:42: 09:25:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.229377 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.229381 | controller |     - '    logger.go:42: 09:25:39 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.229386 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.229390 | controller |       found'
2026-03-14 09:44:27.229394 | controller |     - '    logger.go:42: 09:25:40 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.229398 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.229419 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.229424 | controller |     - '        '
2026-03-14 09:44:27.229428 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.229440 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.229444 | controller |     - '        '
2026-03-14 09:44:27.229449 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.229453 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.229457 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.229461 | controller |     - '          exit 1'
2026-03-14 09:44:27.229466 | controller |     - '        fi'
2026-03-14 09:44:27.229470 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.229476 | controller |     - '        '
2026-03-14 09:44:27.229480 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.229484 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.229488 | controller |       | base64 -d)'
2026-03-14 09:44:27.229493 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.229497 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.229501 | controller |     - '          exit 1'
2026-03-14 09:44:27.229505 | controller |     - '        fi'
2026-03-14 09:44:27.229510 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.229514 | controller |     - '        '
2026-03-14 09:44:27.229518 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.229523 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.229527 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.229531 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.229535 | controller |       --timeout=300s'
2026-03-14 09:44:27.229540 | controller |     - '        '
2026-03-14 09:44:27.229544 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.229548 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.229552 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.229556 | controller |     - '        '
2026-03-14 09:44:27.229561 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.229565 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.229569 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.229573 | controller |     - '        '
2026-03-14 09:44:27.229578 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.229582 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.229594 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.229599 | controller |       available"'
2026-03-14 09:44:27.229603 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.229609 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.229615 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.229632 | controller |     - '          exit 1'
2026-03-14 09:44:27.229639 | controller |     - '        fi'
2026-03-14 09:44:27.229645 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.229650 | controller |     - '        '
2026-03-14 09:44:27.229656 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.229665 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.229671 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.229677 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.229683 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.229689 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.229694 | controller |       -- \'
2026-03-14 09:44:27.229699 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.229705 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.229714 | controller |     - '    logger.go:42: 09:25:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.229719 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.229724 | controller |     - '    logger.go:42: 09:25:40 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.229728 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.229732 | controller |       found'
2026-03-14 09:44:27.229736 | controller |     - '    logger.go:42: 09:25:41 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.229741 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.229765 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.229770 | controller |     - '        '
2026-03-14 09:44:27.229775 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.229779 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.229783 | controller |     - '        '
2026-03-14 09:44:27.229788 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.229792 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.229796 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.229800 | controller |     - '          exit 1'
2026-03-14 09:44:27.229805 | controller |     - '        fi'
2026-03-14 09:44:27.229809 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.229813 | controller |     - '        '
2026-03-14 09:44:27.229817 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.229821 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.229826 | controller |       | base64 -d)'
2026-03-14 09:44:27.229830 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.229834 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.229838 | controller |     - '          exit 1'
2026-03-14 09:44:27.229842 | controller |     - '        fi'
2026-03-14 09:44:27.229847 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.229851 | controller |     - '        '
2026-03-14 09:44:27.229855 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.229859 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.229863 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.229868 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.229872 | controller |       --timeout=300s'
2026-03-14 09:44:27.229876 | controller |     - '        '
2026-03-14 09:44:27.229880 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.229885 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.229891 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.229895 | controller |     - '        '
2026-03-14 09:44:27.229903 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.229907 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.229912 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.229916 | controller |     - '        '
2026-03-14 09:44:27.229920 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.229924 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.229928 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.229933 | controller |       available"'
2026-03-14 09:44:27.229946 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.229951 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.229955 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.229959 | controller |     - '          exit 1'
2026-03-14 09:44:27.229963 | controller |     - '        fi'
2026-03-14 09:44:27.229967 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.229971 | controller |     - '        '
2026-03-14 09:44:27.229976 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.229980 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.229984 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.229988 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.229992 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.229996 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.230001 | controller |       -- \'
2026-03-14 09:44:27.230005 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.230009 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.230013 | controller |     - '    logger.go:42: 09:25:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.230017 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.230022 | controller |     - '    logger.go:42: 09:25:41 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.230026 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.230030 | controller |       found'
2026-03-14 09:44:27.230034 | controller |     - '    logger.go:42: 09:25:42 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.230038 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.230042 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.230047 | controller |     - '        '
2026-03-14 09:44:27.230051 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.230055 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.230059 | controller |     - '        '
2026-03-14 09:44:27.230063 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.230067 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.230072 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.230076 | controller |     - '          exit 1'
2026-03-14 09:44:27.230082 | controller |     - '        fi'
2026-03-14 09:44:27.230086 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.230090 | controller |     - '        '
2026-03-14 09:44:27.230094 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.230098 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.230103 | controller |       | base64 -d)'
2026-03-14 09:44:27.230107 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.230111 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.230115 | controller |     - '          exit 1'
2026-03-14 09:44:27.230119 | controller |     - '        fi'
2026-03-14 09:44:27.230124 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.230128 | controller |     - '        '
2026-03-14 09:44:27.230132 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.230136 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.230143 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.230147 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.230151 | controller |       --timeout=300s'
2026-03-14 09:44:27.230155 | controller |     - '        '
2026-03-14 09:44:27.230160 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.230164 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.230168 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.230172 | controller |     - '        '
2026-03-14 09:44:27.230176 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.230181 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.230185 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.230189 | controller |     - '        '
2026-03-14 09:44:27.230193 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.230197 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.230201 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.230206 | controller |       available"'
2026-03-14 09:44:27.230215 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.230220 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.230224 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.230228 | controller |     - '          exit 1'
2026-03-14 09:44:27.230232 | controller |     - '        fi'
2026-03-14 09:44:27.230236 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.230241 | controller |     - '        '
2026-03-14 09:44:27.230245 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.230249 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.230253 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.230257 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.230262 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.230266 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.230271 | controller |       -- \'
2026-03-14 09:44:27.230276 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.230280 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.230284 | controller |     - '    logger.go:42: 09:25:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.230301 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.230309 | controller |     - '    logger.go:42: 09:25:42 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.230313 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.230317 | controller |       found'
2026-03-14 09:44:27.230322 | controller |     - '    logger.go:42: 09:25:43 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.230326 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.230330 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.230334 | controller |     - '        '
2026-03-14 09:44:27.230338 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.230343 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.230347 | controller |     - '        '
2026-03-14 09:44:27.230351 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.230355 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.230359 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.230364 | controller |     - '          exit 1'
2026-03-14 09:44:27.230368 | controller |     - '        fi'
2026-03-14 09:44:27.230372 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.230376 | controller |     - '        '
2026-03-14 09:44:27.230382 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.230390 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.230405 | controller |       | base64 -d)'
2026-03-14 09:44:27.230411 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.230421 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.230427 | controller |     - '          exit 1'
2026-03-14 09:44:27.230432 | controller |     - '        fi'
2026-03-14 09:44:27.230438 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.230444 | controller |     - '        '
2026-03-14 09:44:27.230450 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.230456 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.230462 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.230467 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.230473 | controller |       --timeout=300s'
2026-03-14 09:44:27.230478 | controller |     - '        '
2026-03-14 09:44:27.230484 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.230489 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.230495 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.230499 | controller |     - '        '
2026-03-14 09:44:27.230503 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.230507 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.230512 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.230518 | controller |     - '        '
2026-03-14 09:44:27.230523 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.230527 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.230531 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.230535 | controller |       available"'
2026-03-14 09:44:27.230539 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.230553 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.230558 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.230562 | controller |     - '          exit 1'
2026-03-14 09:44:27.230566 | controller |     - '        fi'
2026-03-14 09:44:27.230570 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.230574 | controller |     - '        '
2026-03-14 09:44:27.230578 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.230583 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.230587 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.230591 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.230595 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.230599 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.230603 | controller |       -- \'
2026-03-14 09:44:27.230607 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.230612 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.230616 | controller |     - '    logger.go:42: 09:25:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.230620 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.230624 | controller |     - '    logger.go:42: 09:25:43 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.230630 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.230642 | controller |       found'
2026-03-14 09:44:27.230653 | controller |     - '    logger.go:42: 09:25:44 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.230659 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.230665 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.230670 | controller |     - '        '
2026-03-14 09:44:27.230676 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.230682 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.230687 | controller |     - '        '
2026-03-14 09:44:27.230693 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.230698 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.230705 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.230711 | controller |     - '          exit 1'
2026-03-14 09:44:27.230717 | controller |     - '        fi'
2026-03-14 09:44:27.230722 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.230728 | controller |     - '        '
2026-03-14 09:44:27.230738 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.230744 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.230750 | controller |       | base64 -d)'
2026-03-14 09:44:27.230757 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.230762 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.230841 | controller |     - '          exit 1'
2026-03-14 09:44:27.230850 | controller |     - '        fi'
2026-03-14 09:44:27.230856 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.230861 | controller |     - '        '
2026-03-14 09:44:27.230866 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.230872 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.230877 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.230885 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.230890 | controller |       --timeout=300s'
2026-03-14 09:44:27.230897 | controller |     - '        '
2026-03-14 09:44:27.230903 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.230908 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.230913 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.230919 | controller |     - '        '
2026-03-14 09:44:27.230924 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.230929 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.230935 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.230940 | controller |     - '        '
2026-03-14 09:44:27.230945 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.230951 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.230956 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.230962 | controller |       available"'
2026-03-14 09:44:27.230967 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.230972 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.230987 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.230993 | controller |     - '          exit 1'
2026-03-14 09:44:27.230998 | controller |     - '        fi'
2026-03-14 09:44:27.231004 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.231009 | controller |     - '        '
2026-03-14 09:44:27.231015 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.231021 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.231027 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.231032 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.231038 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.231043 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.231049 | controller |       -- \'
2026-03-14 09:44:27.231054 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.231059 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.231065 | controller |     - '    logger.go:42: 09:25:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.231070 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.231078 | controller |     - '    logger.go:42: 09:25:44 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.231086 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.231091 | controller |       found'
2026-03-14 09:44:27.231096 | controller |     - '    logger.go:42: 09:25:45 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.231102 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.231107 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.231113 | controller |     - '        '
2026-03-14 09:44:27.231118 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.231123 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.231129 | controller |     - '        '
2026-03-14 09:44:27.231134 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.231139 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.231145 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.231150 | controller |     - '          exit 1'
2026-03-14 09:44:27.231156 | controller |     - '        fi'
2026-03-14 09:44:27.231161 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.231166 | controller |     - '        '
2026-03-14 09:44:27.231172 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.231177 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.231182 | controller |       | base64 -d)'
2026-03-14 09:44:27.231188 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.231193 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.231199 | controller |     - '          exit 1'
2026-03-14 09:44:27.231206 | controller |     - '        fi'
2026-03-14 09:44:27.231211 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.231217 | controller |     - '        '
2026-03-14 09:44:27.231222 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.231227 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.231233 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.231238 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.231243 | controller |       --timeout=300s'
2026-03-14 09:44:27.231249 | controller |     - '        '
2026-03-14 09:44:27.231254 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.231260 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.231265 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.231270 | controller |     - '        '
2026-03-14 09:44:27.231276 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.231281 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.231304 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.231316 | controller |     - '        '
2026-03-14 09:44:27.231322 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.231328 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.231334 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.231340 | controller |       available"'
2026-03-14 09:44:27.231346 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.231352 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.231360 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.231374 | controller |     - '          exit 1'
2026-03-14 09:44:27.231381 | controller |     - '        fi'
2026-03-14 09:44:27.231387 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.231393 | controller |     - '        '
2026-03-14 09:44:27.231399 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.231405 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.231410 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.231416 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.231422 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.231427 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.231433 | controller |       -- \'
2026-03-14 09:44:27.231438 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.231443 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.231449 | controller |     - '    logger.go:42: 09:25:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.231454 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.231459 | controller |     - '    logger.go:42: 09:25:46 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.231465 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.231470 | controller |       found'
2026-03-14 09:44:27.231475 | controller |     - '    logger.go:42: 09:25:47 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.231481 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.231486 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.231491 | controller |     - '        '
2026-03-14 09:44:27.231497 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.231502 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.231507 | controller |     - '        '
2026-03-14 09:44:27.231512 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.231517 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.231523 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.231528 | controller |     - '          exit 1'
2026-03-14 09:44:27.231534 | controller |     - '        fi'
2026-03-14 09:44:27.231540 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.231546 | controller |     - '        '
2026-03-14 09:44:27.231552 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.231558 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.231563 | controller |       | base64 -d)'
2026-03-14 09:44:27.231569 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.231574 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.231580 | controller |     - '          exit 1'
2026-03-14 09:44:27.231585 | controller |     - '        fi'
2026-03-14 09:44:27.231590 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.231596 | controller |     - '        '
2026-03-14 09:44:27.231601 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.231607 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.231612 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.231620 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.231626 | controller |       --timeout=300s'
2026-03-14 09:44:27.231632 | controller |     - '        '
2026-03-14 09:44:27.231638 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.231643 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.231649 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.231654 | controller |     - '        '
2026-03-14 09:44:27.231659 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.231664 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.231670 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.231675 | controller |     - '        '
2026-03-14 09:44:27.231686 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.231692 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.231698 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.231703 | controller |       available"'
2026-03-14 09:44:27.231709 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.231714 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.231720 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.231726 | controller |     - '          exit 1'
2026-03-14 09:44:27.231732 | controller |     - '        fi'
2026-03-14 09:44:27.231748 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.231755 | controller |     - '        '
2026-03-14 09:44:27.231761 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.231767 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.231773 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.231778 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.231783 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.231788 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.231794 | controller |       -- \'
2026-03-14 09:44:27.231799 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.231804 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.231810 | controller |     - '    logger.go:42: 09:25:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.231815 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.231820 | controller |     - '    logger.go:42: 09:25:47 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.231826 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.231831 | controller |       found'
2026-03-14 09:44:27.231836 | controller |     - '    logger.go:42: 09:25:48 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.231842 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.231847 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.231852 | controller |     - '        '
2026-03-14 09:44:27.231858 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.231865 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.231871 | controller |     - '        '
2026-03-14 09:44:27.231876 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.231881 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.231887 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.231892 | controller |     - '          exit 1'
2026-03-14 09:44:27.231897 | controller |     - '        fi'
2026-03-14 09:44:27.231903 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.231908 | controller |     - '        '
2026-03-14 09:44:27.231914 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.231921 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.231927 | controller |       | base64 -d)'
2026-03-14 09:44:27.231933 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.231940 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.231946 | controller |     - '          exit 1'
2026-03-14 09:44:27.231951 | controller |     - '        fi'
2026-03-14 09:44:27.231956 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.231961 | controller |     - '        '
2026-03-14 09:44:27.231967 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.231972 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.231977 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.231983 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.231988 | controller |       --timeout=300s'
2026-03-14 09:44:27.231993 | controller |     - '        '
2026-03-14 09:44:27.231999 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.232004 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.232009 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.232015 | controller |     - '        '
2026-03-14 09:44:27.232020 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.232025 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.232030 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.232036 | controller |     - '        '
2026-03-14 09:44:27.232041 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.232047 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.232053 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.232058 | controller |       available"'
2026-03-14 09:44:27.232064 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.232070 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.232076 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.232081 | controller |     - '          exit 1'
2026-03-14 09:44:27.232087 | controller |     - '        fi'
2026-03-14 09:44:27.232102 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.232109 | controller |     - '        '
2026-03-14 09:44:27.232115 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.232121 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.232129 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.232136 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.232141 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.232147 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.232154 | controller |       -- \'
2026-03-14 09:44:27.232159 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.232165 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.232174 | controller |     - '    logger.go:42: 09:25:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.232180 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.232186 | controller |     - '    logger.go:42: 09:25:48 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.232192 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.232198 | controller |       found'
2026-03-14 09:44:27.232204 | controller |     - '    logger.go:42: 09:25:49 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.232210 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.232216 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.232221 | controller |     - '        '
2026-03-14 09:44:27.232228 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.232234 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.232239 | controller |     - '        '
2026-03-14 09:44:27.232245 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.232254 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.232260 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.232266 | controller |     - '          exit 1'
2026-03-14 09:44:27.232272 | controller |     - '        fi'
2026-03-14 09:44:27.232278 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.232284 | controller |     - '        '
2026-03-14 09:44:27.232316 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.232323 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.232328 | controller |       | base64 -d)'
2026-03-14 09:44:27.232335 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.232341 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.232346 | controller |     - '          exit 1'
2026-03-14 09:44:27.232352 | controller |     - '        fi'
2026-03-14 09:44:27.232358 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.232364 | controller |     - '        '
2026-03-14 09:44:27.232370 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.232376 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.232382 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.232388 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.232394 | controller |       --timeout=300s'
2026-03-14 09:44:27.232400 | controller |     - '        '
2026-03-14 09:44:27.232406 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.232412 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.232422 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.232431 | controller |     - '        '
2026-03-14 09:44:27.232436 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.232442 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.232447 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.232453 | controller |     - '        '
2026-03-14 09:44:27.232458 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.232464 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.232472 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.232479 | controller |       available"'
2026-03-14 09:44:27.232485 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.232492 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.232498 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.232504 | controller |     - '          exit 1'
2026-03-14 09:44:27.232510 | controller |     - '        fi'
2026-03-14 09:44:27.232516 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.232522 | controller |     - '        '
2026-03-14 09:44:27.232537 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.232544 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.232550 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.232556 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.232562 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.232568 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.232574 | controller |       -- \'
2026-03-14 09:44:27.232580 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.232586 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.232592 | controller |     - '    logger.go:42: 09:25:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.232598 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.232604 | controller |     - '    logger.go:42: 09:25:49 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.232609 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.232615 | controller |       found'
2026-03-14 09:44:27.232621 | controller |     - '    logger.go:42: 09:25:50 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.232626 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.232632 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.232639 | controller |     - '        '
2026-03-14 09:44:27.232645 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.232651 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.232657 | controller |     - '        '
2026-03-14 09:44:27.232674 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.232680 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.232688 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.232693 | controller |     - '          exit 1'
2026-03-14 09:44:27.232699 | controller |     - '        fi'
2026-03-14 09:44:27.232704 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.232712 | controller |     - '        '
2026-03-14 09:44:27.232717 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.232721 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.232725 | controller |       | base64 -d)'
2026-03-14 09:44:27.232730 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.232734 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.232738 | controller |     - '          exit 1'
2026-03-14 09:44:27.232742 | controller |     - '        fi'
2026-03-14 09:44:27.232747 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.232751 | controller |     - '        '
2026-03-14 09:44:27.232755 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.232759 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.232764 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.232768 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.232772 | controller |       --timeout=300s'
2026-03-14 09:44:27.232776 | controller |     - '        '
2026-03-14 09:44:27.232780 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.232785 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.232789 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.232793 | controller |     - '        '
2026-03-14 09:44:27.232797 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.232802 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.232806 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.232810 | controller |     - '        '
2026-03-14 09:44:27.232814 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.232818 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.232823 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.232827 | controller |       available"'
2026-03-14 09:44:27.232831 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.232835 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.232840 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.232844 | controller |     - '          exit 1'
2026-03-14 09:44:27.232848 | controller |     - '        fi'
2026-03-14 09:44:27.232852 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.232857 | controller |     - '        '
2026-03-14 09:44:27.232869 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.232874 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.232878 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.232882 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.232887 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.232891 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.232895 | controller |       -- \'
2026-03-14 09:44:27.232901 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.232906 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.232910 | controller |     - '    logger.go:42: 09:25:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.232914 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.232920 | controller |     - '    logger.go:42: 09:25:50 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.232925 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.232930 | controller |       found'
2026-03-14 09:44:27.232934 | controller |     - '    logger.go:42: 09:25:51 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.232940 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.232945 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.232949 | controller |     - '        '
2026-03-14 09:44:27.232953 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.232957 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.232962 | controller |     - '        '
2026-03-14 09:44:27.232966 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.232970 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.232974 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.232978 | controller |     - '          exit 1'
2026-03-14 09:44:27.232982 | controller |     - '        fi'
2026-03-14 09:44:27.232987 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.232991 | controller |     - '        '
2026-03-14 09:44:27.232995 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.232999 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.233003 | controller |       | base64 -d)'
2026-03-14 09:44:27.233008 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.233012 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.233016 | controller |     - '          exit 1'
2026-03-14 09:44:27.233020 | controller |     - '        fi'
2026-03-14 09:44:27.233024 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.233029 | controller |     - '        '
2026-03-14 09:44:27.233033 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.233037 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.233041 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.233045 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.233049 | controller |       --timeout=300s'
2026-03-14 09:44:27.233054 | controller |     - '        '
2026-03-14 09:44:27.233058 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.233062 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.233066 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.233070 | controller |     - '        '
2026-03-14 09:44:27.233075 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.233079 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.233083 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.233087 | controller |     - '        '
2026-03-14 09:44:27.233091 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.233097 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.233101 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.233105 | controller |       available"'
2026-03-14 09:44:27.233109 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.233115 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.233120 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.233124 | controller |     - '          exit 1'
2026-03-14 09:44:27.233128 | controller |     - '        fi'
2026-03-14 09:44:27.233132 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.233136 | controller |     - '        '
2026-03-14 09:44:27.233141 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.233150 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.233156 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.233160 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.233164 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.233168 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.233173 | controller |       -- \'
2026-03-14 09:44:27.233177 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.233181 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.233185 | controller |     - '    logger.go:42: 09:25:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.233189 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.233194 | controller |     - '    logger.go:42: 09:25:51 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.233198 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.233202 | controller |       found'
2026-03-14 09:44:27.233208 | controller |     - '    logger.go:42: 09:25:52 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.233212 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.233217 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.233221 | controller |     - '        '
2026-03-14 09:44:27.233225 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.233229 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.233234 | controller |     - '        '
2026-03-14 09:44:27.233238 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.233242 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.233246 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.233250 | controller |     - '          exit 1'
2026-03-14 09:44:27.233255 | controller |     - '        fi'
2026-03-14 09:44:27.233259 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.233263 | controller |     - '        '
2026-03-14 09:44:27.233268 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.233272 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.233276 | controller |       | base64 -d)'
2026-03-14 09:44:27.233280 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.233285 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.233309 | controller |     - '          exit 1'
2026-03-14 09:44:27.233314 | controller |     - '        fi'
2026-03-14 09:44:27.233319 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.233323 | controller |     - '        '
2026-03-14 09:44:27.233327 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.233332 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.233336 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.233340 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.233344 | controller |       --timeout=300s'
2026-03-14 09:44:27.233349 | controller |     - '        '
2026-03-14 09:44:27.233353 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.233357 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.233362 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.233366 | controller |     - '        '
2026-03-14 09:44:27.233370 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.233374 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.233378 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.233383 | controller |     - '        '
2026-03-14 09:44:27.233387 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.233391 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.233395 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.233400 | controller |       available"'
2026-03-14 09:44:27.233404 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.233408 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.233412 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.233416 | controller |     - '          exit 1'
2026-03-14 09:44:27.233421 | controller |     - '        fi'
2026-03-14 09:44:27.233425 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.233429 | controller |     - '        '
2026-03-14 09:44:27.233433 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.233438 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.233450 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.233455 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.233460 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.233464 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.233468 | controller |       -- \'
2026-03-14 09:44:27.233472 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.233477 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.233481 | controller |     - '    logger.go:42: 09:25:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.233485 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.233489 | controller |     - '    logger.go:42: 09:25:52 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.233494 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.233499 | controller |       found'
2026-03-14 09:44:27.233504 | controller |     - '    logger.go:42: 09:25:53 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.233508 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.233512 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.233516 | controller |     - '        '
2026-03-14 09:44:27.233521 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.233525 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.233529 | controller |     - '        '
2026-03-14 09:44:27.233534 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.233538 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.233542 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.233546 | controller |     - '          exit 1'
2026-03-14 09:44:27.233551 | controller |     - '        fi'
2026-03-14 09:44:27.233555 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.233559 | controller |     - '        '
2026-03-14 09:44:27.233564 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.233568 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.233572 | controller |       | base64 -d)'
2026-03-14 09:44:27.233576 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.233581 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.233585 | controller |     - '          exit 1'
2026-03-14 09:44:27.233589 | controller |     - '        fi'
2026-03-14 09:44:27.233593 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.233598 | controller |     - '        '
2026-03-14 09:44:27.233602 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.233606 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.233610 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.233616 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.233621 | controller |       --timeout=300s'
2026-03-14 09:44:27.233627 | controller |     - '        '
2026-03-14 09:44:27.233633 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.233639 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.233645 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.233651 | controller |     - '        '
2026-03-14 09:44:27.233660 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.233666 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.233671 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.233677 | controller |     - '        '
2026-03-14 09:44:27.233682 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.233688 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.233692 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.233697 | controller |       available"'
2026-03-14 09:44:27.233702 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.233707 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.233711 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.233717 | controller |     - '          exit 1'
2026-03-14 09:44:27.233721 | controller |     - '        fi'
2026-03-14 09:44:27.233726 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.233730 | controller |     - '        '
2026-03-14 09:44:27.233735 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.233740 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.233752 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.233786 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.233790 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.233794 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.233798 | controller |       -- \'
2026-03-14 09:44:27.233803 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.233807 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.233811 | controller |     - '    logger.go:42: 09:25:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.233816 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.233820 | controller |     - '    logger.go:42: 09:25:54 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.233825 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.233831 | controller |       found'
2026-03-14 09:44:27.233839 | controller |     - '    logger.go:42: 09:25:55 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.233844 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.233850 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.233856 | controller |     - '        '
2026-03-14 09:44:27.233862 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.233868 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.233874 | controller |     - '        '
2026-03-14 09:44:27.233879 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.233885 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.233891 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.233897 | controller |     - '          exit 1'
2026-03-14 09:44:27.233902 | controller |     - '        fi'
2026-03-14 09:44:27.233908 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.233915 | controller |     - '        '
2026-03-14 09:44:27.233924 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.233930 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.233935 | controller |       | base64 -d)'
2026-03-14 09:44:27.233940 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.233944 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.233949 | controller |     - '          exit 1'
2026-03-14 09:44:27.233953 | controller |     - '        fi'
2026-03-14 09:44:27.233957 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.233962 | controller |     - '        '
2026-03-14 09:44:27.233966 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.233970 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.233975 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.233979 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.233985 | controller |       --timeout=300s'
2026-03-14 09:44:27.233990 | controller |     - '        '
2026-03-14 09:44:27.233994 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.233998 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.234002 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.234007 | controller |     - '        '
2026-03-14 09:44:27.234011 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.234015 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.234019 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.234024 | controller |     - '        '
2026-03-14 09:44:27.234028 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.234032 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.234037 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.234041 | controller |       available"'
2026-03-14 09:44:27.234045 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.234050 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.234054 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.234058 | controller |     - '          exit 1'
2026-03-14 09:44:27.234062 | controller |     - '        fi'
2026-03-14 09:44:27.234067 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.234071 | controller |     - '        '
2026-03-14 09:44:27.234075 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.234079 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.234092 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.234097 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.234101 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.234105 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.234110 | controller |       -- \'
2026-03-14 09:44:27.234117 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.234121 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.234126 | controller |     - '    logger.go:42: 09:25:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.234130 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.234134 | controller |     - '    logger.go:42: 09:25:55 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.234139 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.234143 | controller |       found'
2026-03-14 09:44:27.234149 | controller |     - '    logger.go:42: 09:25:56 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.234153 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.234157 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.234162 | controller |     - '        '
2026-03-14 09:44:27.234166 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.234170 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.234176 | controller |     - '        '
2026-03-14 09:44:27.234180 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.234185 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.234189 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.234193 | controller |     - '          exit 1'
2026-03-14 09:44:27.234197 | controller |     - '        fi'
2026-03-14 09:44:27.234202 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.234206 | controller |     - '        '
2026-03-14 09:44:27.234210 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.234214 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.234219 | controller |       | base64 -d)'
2026-03-14 09:44:27.234223 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.234227 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.234232 | controller |     - '          exit 1'
2026-03-14 09:44:27.234236 | controller |     - '        fi'
2026-03-14 09:44:27.234240 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.234245 | controller |     - '        '
2026-03-14 09:44:27.234249 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.234253 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.234257 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.234262 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.234266 | controller |       --timeout=300s'
2026-03-14 09:44:27.234270 | controller |     - '        '
2026-03-14 09:44:27.234274 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.234279 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.234283 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.234300 | controller |     - '        '
2026-03-14 09:44:27.234308 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.234313 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.234317 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.234321 | controller |     - '        '
2026-03-14 09:44:27.234326 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.234330 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.234334 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.234339 | controller |       available"'
2026-03-14 09:44:27.234343 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.234347 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.234351 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.234356 | controller |     - '          exit 1'
2026-03-14 09:44:27.234360 | controller |     - '        fi'
2026-03-14 09:44:27.234364 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.234369 | controller |     - '        '
2026-03-14 09:44:27.234373 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.234377 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.234381 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.234395 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.234401 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.234405 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.234409 | controller |       -- \'
2026-03-14 09:44:27.234413 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.234418 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.234422 | controller |     - '    logger.go:42: 09:25:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.234426 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.234430 | controller |     - '    logger.go:42: 09:25:56 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.234434 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.234439 | controller |       found'
2026-03-14 09:44:27.234443 | controller |     - '    logger.go:42: 09:25:57 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.234447 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.234451 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.234456 | controller |     - '        '
2026-03-14 09:44:27.234460 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.234464 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.234468 | controller |     - '        '
2026-03-14 09:44:27.234472 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.234476 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.234481 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.234485 | controller |     - '          exit 1'
2026-03-14 09:44:27.234489 | controller |     - '        fi'
2026-03-14 09:44:27.234493 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.234497 | controller |     - '        '
2026-03-14 09:44:27.234502 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.234506 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.234510 | controller |       | base64 -d)'
2026-03-14 09:44:27.234514 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.234519 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.234523 | controller |     - '          exit 1'
2026-03-14 09:44:27.234527 | controller |     - '        fi'
2026-03-14 09:44:27.234531 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.234535 | controller |     - '        '
2026-03-14 09:44:27.234540 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.234544 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.234548 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.234552 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.234556 | controller |       --timeout=300s'
2026-03-14 09:44:27.234560 | controller |     - '        '
2026-03-14 09:44:27.234565 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.234569 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.234573 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.234579 | controller |     - '        '
2026-03-14 09:44:27.234583 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.234589 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.234594 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.234598 | controller |     - '        '
2026-03-14 09:44:27.234602 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.234606 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.234610 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.234616 | controller |       available"'
2026-03-14 09:44:27.234621 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.234627 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.234633 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.234639 | controller |     - '          exit 1'
2026-03-14 09:44:27.234645 | controller |     - '        fi'
2026-03-14 09:44:27.234651 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.234657 | controller |     - '        '
2026-03-14 09:44:27.234662 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.234668 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.234673 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.234679 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.234692 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.234697 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.234701 | controller |       -- \'
2026-03-14 09:44:27.234706 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.234710 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.234714 | controller |     - '    logger.go:42: 09:25:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.234718 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.234723 | controller |     - '    logger.go:42: 09:25:57 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.234727 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.234731 | controller |       found'
2026-03-14 09:44:27.234738 | controller |     - '    logger.go:42: 09:25:58 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.234742 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.234747 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.234751 | controller |     - '        '
2026-03-14 09:44:27.234756 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.234760 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.234764 | controller |     - '        '
2026-03-14 09:44:27.234769 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.234773 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.234777 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.234781 | controller |     - '          exit 1'
2026-03-14 09:44:27.234786 | controller |     - '        fi'
2026-03-14 09:44:27.234790 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.234794 | controller |     - '        '
2026-03-14 09:44:27.234800 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.234805 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.234809 | controller |       | base64 -d)'
2026-03-14 09:44:27.234813 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.234818 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.234822 | controller |     - '          exit 1'
2026-03-14 09:44:27.234826 | controller |     - '        fi'
2026-03-14 09:44:27.234830 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.234835 | controller |     - '        '
2026-03-14 09:44:27.234843 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.234848 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.234852 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.234856 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.234861 | controller |       --timeout=300s'
2026-03-14 09:44:27.234865 | controller |     - '        '
2026-03-14 09:44:27.234869 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.234874 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.234878 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.234882 | controller |     - '        '
2026-03-14 09:44:27.234886 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.234891 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.234895 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.234899 | controller |     - '        '
2026-03-14 09:44:27.234904 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.234908 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.234914 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.234919 | controller |       available"'
2026-03-14 09:44:27.234923 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.234928 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.234933 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.234938 | controller |     - '          exit 1'
2026-03-14 09:44:27.234943 | controller |     - '        fi'
2026-03-14 09:44:27.234947 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.234951 | controller |     - '        '
2026-03-14 09:44:27.234956 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.234960 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.234964 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.234968 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.234981 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.234985 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.234990 | controller |       -- \'
2026-03-14 09:44:27.234994 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.235000 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.235004 | controller |     - '    logger.go:42: 09:25:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.235008 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.235013 | controller |     - '    logger.go:42: 09:25:58 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.235017 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.235021 | controller |       found'
2026-03-14 09:44:27.235027 | controller |     - '    logger.go:42: 09:25:59 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.235031 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.235035 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.235039 | controller |     - '        '
2026-03-14 09:44:27.235044 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.235048 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.235052 | controller |     - '        '
2026-03-14 09:44:27.235056 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.235060 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.235065 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.235069 | controller |     - '          exit 1'
2026-03-14 09:44:27.235073 | controller |     - '        fi'
2026-03-14 09:44:27.235077 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.235082 | controller |     - '        '
2026-03-14 09:44:27.235086 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.235090 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.235094 | controller |       | base64 -d)'
2026-03-14 09:44:27.235098 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.235104 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.235108 | controller |     - '          exit 1'
2026-03-14 09:44:27.235113 | controller |     - '        fi'
2026-03-14 09:44:27.235117 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.235121 | controller |     - '        '
2026-03-14 09:44:27.235125 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.235129 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.235134 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.235138 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.235142 | controller |       --timeout=300s'
2026-03-14 09:44:27.235146 | controller |     - '        '
2026-03-14 09:44:27.235150 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.235155 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.235159 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.235163 | controller |     - '        '
2026-03-14 09:44:27.235167 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.235171 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.235176 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.235180 | controller |     - '        '
2026-03-14 09:44:27.235184 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.235190 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.235194 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.235198 | controller |       available"'
2026-03-14 09:44:27.235202 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.235208 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.235212 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.235216 | controller |     - '          exit 1'
2026-03-14 09:44:27.235220 | controller |     - '        fi'
2026-03-14 09:44:27.235225 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.235229 | controller |     - '        '
2026-03-14 09:44:27.235233 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.235237 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.235241 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.235245 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.235256 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.235260 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.235265 | controller |       -- \'
2026-03-14 09:44:27.235269 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.235273 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.235277 | controller |     - '    logger.go:42: 09:25:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.235282 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.235286 | controller |     - '    logger.go:42: 09:25:59 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.235306 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.235310 | controller |       found'
2026-03-14 09:44:27.235315 | controller |     - '    logger.go:42: 09:26:00 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.235319 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.235323 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.235327 | controller |     - '        '
2026-03-14 09:44:27.235332 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.235336 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.235340 | controller |     - '        '
2026-03-14 09:44:27.235344 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.235348 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.235353 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.235357 | controller |     - '          exit 1'
2026-03-14 09:44:27.235361 | controller |     - '        fi'
2026-03-14 09:44:27.235365 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.235369 | controller |     - '        '
2026-03-14 09:44:27.235374 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.235378 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.235382 | controller |       | base64 -d)'
2026-03-14 09:44:27.235386 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.235391 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.235395 | controller |     - '          exit 1'
2026-03-14 09:44:27.235399 | controller |     - '        fi'
2026-03-14 09:44:27.235405 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.235409 | controller |     - '        '
2026-03-14 09:44:27.235414 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.235418 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.235422 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.235426 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.235430 | controller |       --timeout=300s'
2026-03-14 09:44:27.235437 | controller |     - '        '
2026-03-14 09:44:27.235441 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.235446 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.235450 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.235454 | controller |     - '        '
2026-03-14 09:44:27.235458 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.235463 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.235467 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.235471 | controller |     - '        '
2026-03-14 09:44:27.235475 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.235479 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.235483 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.235488 | controller |       available"'
2026-03-14 09:44:27.235492 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.235496 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.235500 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.235505 | controller |     - '          exit 1'
2026-03-14 09:44:27.235509 | controller |     - '        fi'
2026-03-14 09:44:27.235513 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.235518 | controller |     - '        '
2026-03-14 09:44:27.235522 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.235526 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.235530 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.235535 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.235539 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.235550 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.235555 | controller |       -- \'
2026-03-14 09:44:27.235559 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.235563 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.235568 | controller |     - '    logger.go:42: 09:26:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.235572 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.235578 | controller |     - '    logger.go:42: 09:26:00 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.235582 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.235588 | controller |       found'
2026-03-14 09:44:27.235594 | controller |     - '    logger.go:42: 09:26:01 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.235599 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.235605 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.235609 | controller |     - '        '
2026-03-14 09:44:27.235614 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.235620 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.235626 | controller |     - '        '
2026-03-14 09:44:27.235632 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.235638 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.235644 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.235650 | controller |     - '          exit 1'
2026-03-14 09:44:27.235656 | controller |     - '        fi'
2026-03-14 09:44:27.235662 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.235668 | controller |     - '        '
2026-03-14 09:44:27.235673 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.235678 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.235684 | controller |       | base64 -d)'
2026-03-14 09:44:27.235690 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.235694 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.235698 | controller |     - '          exit 1'
2026-03-14 09:44:27.235703 | controller |     - '        fi'
2026-03-14 09:44:27.235707 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.235711 | controller |     - '        '
2026-03-14 09:44:27.235715 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.235719 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.235724 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.235728 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.235732 | controller |       --timeout=300s'
2026-03-14 09:44:27.235736 | controller |     - '        '
2026-03-14 09:44:27.235740 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.235745 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.235749 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.235753 | controller |     - '        '
2026-03-14 09:44:27.235757 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.235761 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.235765 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.235770 | controller |     - '        '
2026-03-14 09:44:27.235774 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.235778 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.235782 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.235786 | controller |       available"'
2026-03-14 09:44:27.235791 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.235795 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.235799 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.235805 | controller |     - '          exit 1'
2026-03-14 09:44:27.235809 | controller |     - '        fi'
2026-03-14 09:44:27.235814 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.235818 | controller |     - '        '
2026-03-14 09:44:27.235822 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.235826 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.235830 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.235835 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.235839 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.235850 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.235855 | controller |       -- \'
2026-03-14 09:44:27.235859 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.235864 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.235868 | controller |     - '    logger.go:42: 09:26:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.235872 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.235876 | controller |     - '    logger.go:42: 09:26:02 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.235880 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.235885 | controller |       found'
2026-03-14 09:44:27.235889 | controller |     - '    logger.go:42: 09:26:03 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.235893 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.235897 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.235902 | controller |     - '        '
2026-03-14 09:44:27.235906 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.235910 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.235914 | controller |     - '        '
2026-03-14 09:44:27.235918 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.235923 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.235927 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.235931 | controller |     - '          exit 1'
2026-03-14 09:44:27.235935 | controller |     - '        fi'
2026-03-14 09:44:27.235942 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.235946 | controller |     - '        '
2026-03-14 09:44:27.235950 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.235955 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.235959 | controller |       | base64 -d)'
2026-03-14 09:44:27.235963 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.235967 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.235971 | controller |     - '          exit 1'
2026-03-14 09:44:27.235975 | controller |     - '        fi'
2026-03-14 09:44:27.235980 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.235984 | controller |     - '        '
2026-03-14 09:44:27.235988 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.235992 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.235996 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.236001 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.236011 | controller |       --timeout=300s'
2026-03-14 09:44:27.236015 | controller |     - '        '
2026-03-14 09:44:27.236019 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.236024 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.236028 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.236032 | controller |     - '        '
2026-03-14 09:44:27.236036 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.236040 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.236045 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.236049 | controller |     - '        '
2026-03-14 09:44:27.236053 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.236057 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.236061 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.236065 | controller |       available"'
2026-03-14 09:44:27.236070 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.236074 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.236078 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.236082 | controller |     - '          exit 1'
2026-03-14 09:44:27.236086 | controller |     - '        fi'
2026-03-14 09:44:27.236090 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.236095 | controller |     - '        '
2026-03-14 09:44:27.236099 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.236103 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.236107 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.236111 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.236116 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.236120 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.236124 | controller |       -- \'
2026-03-14 09:44:27.236134 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.236139 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.236143 | controller |     - '    logger.go:42: 09:26:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.236147 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.236152 | controller |     - '    logger.go:42: 09:26:03 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.236158 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.236162 | controller |       found'
2026-03-14 09:44:27.236166 | controller |     - '    logger.go:42: 09:26:04 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.236170 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.236175 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.236179 | controller |     - '        '
2026-03-14 09:44:27.236183 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.236187 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.236194 | controller |     - '        '
2026-03-14 09:44:27.236198 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.236202 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.236207 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.236211 | controller |     - '          exit 1'
2026-03-14 09:44:27.236215 | controller |     - '        fi'
2026-03-14 09:44:27.236219 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.236224 | controller |     - '        '
2026-03-14 09:44:27.236228 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.236232 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.236236 | controller |       | base64 -d)'
2026-03-14 09:44:27.236240 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.236244 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.236249 | controller |     - '          exit 1'
2026-03-14 09:44:27.236253 | controller |     - '        fi'
2026-03-14 09:44:27.236257 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.236261 | controller |     - '        '
2026-03-14 09:44:27.236265 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.236270 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.236274 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.236278 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.236282 | controller |       --timeout=300s'
2026-03-14 09:44:27.236289 | controller |     - '        '
2026-03-14 09:44:27.236308 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.236313 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.236317 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.236321 | controller |     - '        '
2026-03-14 09:44:27.236325 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.236330 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.236334 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.236338 | controller |     - '        '
2026-03-14 09:44:27.236342 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.236346 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.236351 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.236355 | controller |       available"'
2026-03-14 09:44:27.236359 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.236363 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.236367 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.236372 | controller |     - '          exit 1'
2026-03-14 09:44:27.236376 | controller |     - '        fi'
2026-03-14 09:44:27.236380 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.236384 | controller |     - '        '
2026-03-14 09:44:27.236388 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.236393 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.236397 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.236403 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.236407 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.236411 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.236415 | controller |       -- \'
2026-03-14 09:44:27.236426 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.236430 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.236436 | controller |     - '    logger.go:42: 09:26:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.236441 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.236445 | controller |     - '    logger.go:42: 09:26:04 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.236449 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.236455 | controller |       found'
2026-03-14 09:44:27.236461 | controller |     - '    logger.go:42: 09:26:05 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.236467 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.236475 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.236482 | controller |     - '        '
2026-03-14 09:44:27.236488 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.236494 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.236500 | controller |     - '        '
2026-03-14 09:44:27.236506 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.236511 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.236517 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.236523 | controller |     - '          exit 1'
2026-03-14 09:44:27.236529 | controller |     - '        fi'
2026-03-14 09:44:27.236535 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.236542 | controller |     - '        '
2026-03-14 09:44:27.236550 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.236557 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.236563 | controller |       | base64 -d)'
2026-03-14 09:44:27.236569 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.236574 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.236580 | controller |     - '          exit 1'
2026-03-14 09:44:27.236587 | controller |     - '        fi'
2026-03-14 09:44:27.236593 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.236599 | controller |     - '        '
2026-03-14 09:44:27.236605 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.236611 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.236618 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.236624 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.236630 | controller |       --timeout=300s'
2026-03-14 09:44:27.236636 | controller |     - '        '
2026-03-14 09:44:27.236642 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.236648 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.236653 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.236659 | controller |     - '        '
2026-03-14 09:44:27.236665 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.236673 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.236681 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.236687 | controller |     - '        '
2026-03-14 09:44:27.236691 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.236695 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.236700 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.236704 | controller |       available"'
2026-03-14 09:44:27.236708 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.236712 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.236716 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.236720 | controller |     - '          exit 1'
2026-03-14 09:44:27.236725 | controller |     - '        fi'
2026-03-14 09:44:27.236729 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.236733 | controller |     - '        '
2026-03-14 09:44:27.236737 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.236741 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.236746 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.236750 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.236754 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.236758 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.236762 | controller |       -- \'
2026-03-14 09:44:27.236767 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.236780 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.236785 | controller |     - '    logger.go:42: 09:26:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.236789 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.236793 | controller |     - '    logger.go:42: 09:26:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.236797 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.236802 | controller |       found'
2026-03-14 09:44:27.236808 | controller |     - '    logger.go:42: 09:26:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.236812 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.236816 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.236820 | controller |     - '        '
2026-03-14 09:44:27.236824 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.236829 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.236835 | controller |     - '        '
2026-03-14 09:44:27.236841 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.236847 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.236852 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.236858 | controller |     - '          exit 1'
2026-03-14 09:44:27.236864 | controller |     - '        fi'
2026-03-14 09:44:27.236870 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.236875 | controller |     - '        '
2026-03-14 09:44:27.236881 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.236890 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.236895 | controller |       | base64 -d)'
2026-03-14 09:44:27.236901 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.236907 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.236912 | controller |     - '          exit 1'
2026-03-14 09:44:27.236918 | controller |     - '        fi'
2026-03-14 09:44:27.236922 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.236927 | controller |     - '        '
2026-03-14 09:44:27.236931 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.236935 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.236939 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.236943 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.236947 | controller |       --timeout=300s'
2026-03-14 09:44:27.236952 | controller |     - '        '
2026-03-14 09:44:27.236956 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.236960 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.236964 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.236968 | controller |     - '        '
2026-03-14 09:44:27.236973 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.236977 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.236981 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.236985 | controller |     - '        '
2026-03-14 09:44:27.236989 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.236994 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.236998 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.237002 | controller |       available"'
2026-03-14 09:44:27.237006 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.237010 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.237014 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.237019 | controller |     - '          exit 1'
2026-03-14 09:44:27.237023 | controller |     - '        fi'
2026-03-14 09:44:27.237030 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.237034 | controller |     - '        '
2026-03-14 09:44:27.237039 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.237043 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.237047 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.237051 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.237055 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.237059 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.237064 | controller |       -- \'
2026-03-14 09:44:27.237068 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.237082 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.237086 | controller |     - '    logger.go:42: 09:26:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.237091 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.237095 | controller |     - '    logger.go:42: 09:26:06 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.237099 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.237103 | controller |       found'
2026-03-14 09:44:27.237108 | controller |     - '    logger.go:42: 09:26:07 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.237112 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.237116 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.237120 | controller |     - '        '
2026-03-14 09:44:27.237124 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.237128 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.237133 | controller |     - '        '
2026-03-14 09:44:27.237137 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.237141 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.237145 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.237149 | controller |     - '          exit 1'
2026-03-14 09:44:27.237154 | controller |     - '        fi'
2026-03-14 09:44:27.237158 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.237162 | controller |     - '        '
2026-03-14 09:44:27.237166 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.237171 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.237175 | controller |       | base64 -d)'
2026-03-14 09:44:27.237179 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.237183 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.237188 | controller |     - '          exit 1'
2026-03-14 09:44:27.237192 | controller |     - '        fi'
2026-03-14 09:44:27.237196 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.237200 | controller |     - '        '
2026-03-14 09:44:27.237205 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.237209 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.237213 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.237217 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.237222 | controller |       --timeout=300s'
2026-03-14 09:44:27.237226 | controller |     - '        '
2026-03-14 09:44:27.237230 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.237235 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.237239 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.237243 | controller |     - '        '
2026-03-14 09:44:27.237247 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.237252 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.237256 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.237260 | controller |     - '        '
2026-03-14 09:44:27.237264 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.237269 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.237274 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.237281 | controller |       available"'
2026-03-14 09:44:27.237286 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.237311 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.237316 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.237321 | controller |     - '          exit 1'
2026-03-14 09:44:27.237325 | controller |     - '        fi'
2026-03-14 09:44:27.237329 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.237334 | controller |     - '        '
2026-03-14 09:44:27.237338 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.237342 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.237346 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.237351 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.237355 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.237359 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.237363 | controller |       -- \'
2026-03-14 09:44:27.237368 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.237372 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.237383 | controller |     - '    logger.go:42: 09:26:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.237388 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.237392 | controller |     - '    logger.go:42: 09:26:07 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.237396 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.237400 | controller |       found'
2026-03-14 09:44:27.237405 | controller |     - '    logger.go:42: 09:26:08 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.237409 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.237413 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.237418 | controller |     - '        '
2026-03-14 09:44:27.237424 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.237428 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.237433 | controller |     - '        '
2026-03-14 09:44:27.237437 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.237441 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.237445 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.237450 | controller |     - '          exit 1'
2026-03-14 09:44:27.237454 | controller |     - '        fi'
2026-03-14 09:44:27.237458 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.237463 | controller |     - '        '
2026-03-14 09:44:27.237467 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.237471 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.237475 | controller |       | base64 -d)'
2026-03-14 09:44:27.237480 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.237484 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.237488 | controller |     - '          exit 1'
2026-03-14 09:44:27.237493 | controller |     - '        fi'
2026-03-14 09:44:27.237497 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.237504 | controller |     - '        '
2026-03-14 09:44:27.237508 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.237512 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.237516 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.237521 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.237525 | controller |       --timeout=300s'
2026-03-14 09:44:27.237529 | controller |     - '        '
2026-03-14 09:44:27.237533 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.237537 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.237542 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.237546 | controller |     - '        '
2026-03-14 09:44:27.237550 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.237554 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.237559 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.237563 | controller |     - '        '
2026-03-14 09:44:27.237567 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.237571 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.237575 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.237579 | controller |       available"'
2026-03-14 09:44:27.237584 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.237588 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.237592 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.237596 | controller |     - '          exit 1'
2026-03-14 09:44:27.237600 | controller |     - '        fi'
2026-03-14 09:44:27.237605 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.237609 | controller |     - '        '
2026-03-14 09:44:27.237614 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.237620 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.237625 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.237635 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.237648 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.237654 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.237660 | controller |       -- \'
2026-03-14 09:44:27.237666 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.237672 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.237695 | controller |     - '    logger.go:42: 09:26:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.237702 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.237707 | controller |     - '    logger.go:42: 09:26:08 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.237713 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.237719 | controller |       found'
2026-03-14 09:44:27.237727 | controller |     - '    logger.go:42: 09:26:09 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.237732 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.237738 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.237744 | controller |     - '        '
2026-03-14 09:44:27.237750 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.237755 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.237780 | controller |     - '        '
2026-03-14 09:44:27.237786 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.237792 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.237798 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.237804 | controller |     - '          exit 1'
2026-03-14 09:44:27.237810 | controller |     - '        fi'
2026-03-14 09:44:27.237816 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.237821 | controller |     - '        '
2026-03-14 09:44:27.237827 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.237833 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.237838 | controller |       | base64 -d)'
2026-03-14 09:44:27.237844 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.237849 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.237854 | controller |     - '          exit 1'
2026-03-14 09:44:27.237860 | controller |     - '        fi'
2026-03-14 09:44:27.237866 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.237872 | controller |     - '        '
2026-03-14 09:44:27.237878 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.237884 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.237890 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.237896 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.237902 | controller |       --timeout=300s'
2026-03-14 09:44:27.237908 | controller |     - '        '
2026-03-14 09:44:27.237914 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.237919 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.237925 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.237929 | controller |     - '        '
2026-03-14 09:44:27.237936 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.237940 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.237944 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.237949 | controller |     - '        '
2026-03-14 09:44:27.237953 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.237959 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.237965 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.237971 | controller |       available"'
2026-03-14 09:44:27.237977 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.237985 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.237991 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.237997 | controller |     - '          exit 1'
2026-03-14 09:44:27.238003 | controller |     - '        fi'
2026-03-14 09:44:27.238011 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.238016 | controller |     - '        '
2026-03-14 09:44:27.238022 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.238028 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.238034 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.238040 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.238045 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.238051 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.238057 | controller |       -- \'
2026-03-14 09:44:27.238062 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.238067 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.238073 | controller |     - '    logger.go:42: 09:26:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.238079 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.238094 | controller |     - '    logger.go:42: 09:26:10 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.238101 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.238107 | controller |       found'
2026-03-14 09:44:27.238113 | controller |     - '    logger.go:42: 09:26:11 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.238119 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.238124 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.238130 | controller |     - '        '
2026-03-14 09:44:27.238136 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.238144 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.238150 | controller |     - '        '
2026-03-14 09:44:27.238156 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.238162 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.238168 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.238174 | controller |     - '          exit 1'
2026-03-14 09:44:27.238179 | controller |     - '        fi'
2026-03-14 09:44:27.238185 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.238191 | controller |     - '        '
2026-03-14 09:44:27.238197 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.238203 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.238209 | controller |       | base64 -d)'
2026-03-14 09:44:27.238214 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.238220 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.238226 | controller |     - '          exit 1'
2026-03-14 09:44:27.238232 | controller |     - '        fi'
2026-03-14 09:44:27.238238 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.238243 | controller |     - '        '
2026-03-14 09:44:27.238249 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.238255 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.238261 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.238267 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.238273 | controller |       --timeout=300s'
2026-03-14 09:44:27.238279 | controller |     - '        '
2026-03-14 09:44:27.238307 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.238316 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.238322 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.238328 | controller |     - '        '
2026-03-14 09:44:27.238334 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.238340 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.238345 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.238351 | controller |     - '        '
2026-03-14 09:44:27.238357 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.238363 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.238368 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.238372 | controller |       available"'
2026-03-14 09:44:27.238376 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.238381 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.238385 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.238389 | controller |     - '          exit 1'
2026-03-14 09:44:27.238393 | controller |     - '        fi'
2026-03-14 09:44:27.238398 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.238402 | controller |     - '        '
2026-03-14 09:44:27.238406 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.238410 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.238415 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.238419 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.238423 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.238427 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.238432 | controller |       -- \'
2026-03-14 09:44:27.238436 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.238440 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.238444 | controller |     - '    logger.go:42: 09:26:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.238449 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.238461 | controller |     - '    logger.go:42: 09:26:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.238466 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.238470 | controller |       found'
2026-03-14 09:44:27.238474 | controller |     - '    logger.go:42: 09:26:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.238478 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.238483 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.238487 | controller |     - '        '
2026-03-14 09:44:27.238491 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.238495 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.238500 | controller |     - '        '
2026-03-14 09:44:27.238504 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.238510 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.238514 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.238518 | controller |     - '          exit 1'
2026-03-14 09:44:27.238523 | controller |     - '        fi'
2026-03-14 09:44:27.238527 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.238531 | controller |     - '        '
2026-03-14 09:44:27.238535 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.238539 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.238544 | controller |       | base64 -d)'
2026-03-14 09:44:27.238548 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.238552 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.238556 | controller |     - '          exit 1'
2026-03-14 09:44:27.238560 | controller |     - '        fi'
2026-03-14 09:44:27.238565 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.238569 | controller |     - '        '
2026-03-14 09:44:27.238573 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.238577 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.238581 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.238585 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.238590 | controller |       --timeout=300s'
2026-03-14 09:44:27.238594 | controller |     - '        '
2026-03-14 09:44:27.238598 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.238602 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.238606 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.238611 | controller |     - '        '
2026-03-14 09:44:27.238615 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.238619 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.238623 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.238627 | controller |     - '        '
2026-03-14 09:44:27.238632 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.238636 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.238640 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.238644 | controller |       available"'
2026-03-14 09:44:27.238648 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.238652 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.238657 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.238661 | controller |     - '          exit 1'
2026-03-14 09:44:27.238665 | controller |     - '        fi'
2026-03-14 09:44:27.238669 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.238674 | controller |     - '        '
2026-03-14 09:44:27.238678 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.238682 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.238686 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.238691 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.238696 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.238701 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.238705 | controller |       -- \'
2026-03-14 09:44:27.238709 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.238713 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.238718 | controller |     - '    logger.go:42: 09:26:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.238723 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.238727 | controller |     - '    logger.go:42: 09:26:12 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.238737 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.238741 | controller |       found'
2026-03-14 09:44:27.238746 | controller |     - '    logger.go:42: 09:26:13 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.238750 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.238754 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.238759 | controller |     - '        '
2026-03-14 09:44:27.238763 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.238767 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.238771 | controller |     - '        '
2026-03-14 09:44:27.238778 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.238783 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.238787 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.238791 | controller |     - '          exit 1'
2026-03-14 09:44:27.238796 | controller |     - '        fi'
2026-03-14 09:44:27.238800 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.238804 | controller |     - '        '
2026-03-14 09:44:27.238808 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.238813 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.238817 | controller |       | base64 -d)'
2026-03-14 09:44:27.238821 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.238825 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.238830 | controller |     - '          exit 1'
2026-03-14 09:44:27.238834 | controller |     - '        fi'
2026-03-14 09:44:27.238838 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.238842 | controller |     - '        '
2026-03-14 09:44:27.238846 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.238850 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.238855 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.238859 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.238863 | controller |       --timeout=300s'
2026-03-14 09:44:27.238867 | controller |     - '        '
2026-03-14 09:44:27.238871 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.238875 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.238880 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.238884 | controller |     - '        '
2026-03-14 09:44:27.238888 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.238894 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.238898 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.238902 | controller |     - '        '
2026-03-14 09:44:27.238906 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.238912 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.238916 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.238920 | controller |       available"'
2026-03-14 09:44:27.238924 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.238929 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.238933 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.238937 | controller |     - '          exit 1'
2026-03-14 09:44:27.238941 | controller |     - '        fi'
2026-03-14 09:44:27.238945 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.238950 | controller |     - '        '
2026-03-14 09:44:27.238954 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.238958 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.238962 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.238966 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.238970 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.238975 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.238979 | controller |       -- \'
2026-03-14 09:44:27.238983 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.238987 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.238991 | controller |     - '    logger.go:42: 09:26:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.238995 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.239000 | controller |     - '    logger.go:42: 09:26:13 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.239009 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.239014 | controller |       found'
2026-03-14 09:44:27.239018 | controller |     - '    logger.go:42: 09:26:14 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.239022 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.239027 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.239031 | controller |     - '        '
2026-03-14 09:44:27.239035 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.239039 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.239043 | controller |     - '        '
2026-03-14 09:44:27.239048 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.239052 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.239056 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.239060 | controller |     - '          exit 1'
2026-03-14 09:44:27.239064 | controller |     - '        fi'
2026-03-14 09:44:27.239068 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.239073 | controller |     - '        '
2026-03-14 09:44:27.239077 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.239081 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.239087 | controller |       | base64 -d)'
2026-03-14 09:44:27.239091 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.239095 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.239099 | controller |     - '          exit 1'
2026-03-14 09:44:27.239104 | controller |     - '        fi'
2026-03-14 09:44:27.239108 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.239112 | controller |     - '        '
2026-03-14 09:44:27.239116 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.239120 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.239125 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.239129 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.239133 | controller |       --timeout=300s'
2026-03-14 09:44:27.239137 | controller |     - '        '
2026-03-14 09:44:27.239141 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.239146 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.239150 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.239154 | controller |     - '        '
2026-03-14 09:44:27.239158 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.239162 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.239166 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.239170 | controller |     - '        '
2026-03-14 09:44:27.239175 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.239179 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.239183 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.239187 | controller |       available"'
2026-03-14 09:44:27.239191 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.239196 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.239200 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.239204 | controller |     - '          exit 1'
2026-03-14 09:44:27.239208 | controller |     - '        fi'
2026-03-14 09:44:27.239212 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.239216 | controller |     - '        '
2026-03-14 09:44:27.239221 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.239225 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.239229 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.239233 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.239237 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.239241 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.239246 | controller |       -- \'
2026-03-14 09:44:27.239250 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.239254 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.239260 | controller |     - '    logger.go:42: 09:26:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.239264 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.239268 | controller |     - '    logger.go:42: 09:26:14 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.239272 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.239277 | controller |       found'
2026-03-14 09:44:27.239286 | controller |     - '    logger.go:42: 09:26:15 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.239306 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.239310 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.239314 | controller |     - '        '
2026-03-14 09:44:27.239319 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.239323 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.239327 | controller |     - '        '
2026-03-14 09:44:27.239331 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.239335 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.239340 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.239344 | controller |     - '          exit 1'
2026-03-14 09:44:27.239348 | controller |     - '        fi'
2026-03-14 09:44:27.239352 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.239361 | controller |     - '        '
2026-03-14 09:44:27.239365 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.239370 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.239374 | controller |       | base64 -d)'
2026-03-14 09:44:27.239379 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.239385 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.239389 | controller |     - '          exit 1'
2026-03-14 09:44:27.239393 | controller |     - '        fi'
2026-03-14 09:44:27.239397 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.239402 | controller |     - '        '
2026-03-14 09:44:27.239406 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.239411 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.239416 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.239420 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.239424 | controller |       --timeout=300s'
2026-03-14 09:44:27.239428 | controller |     - '        '
2026-03-14 09:44:27.239432 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.239436 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.239441 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.239445 | controller |     - '        '
2026-03-14 09:44:27.239449 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.239453 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.239458 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.239462 | controller |     - '        '
2026-03-14 09:44:27.239466 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.239470 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.239476 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.239480 | controller |       available"'
2026-03-14 09:44:27.239484 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.239489 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.239493 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.239497 | controller |     - '          exit 1'
2026-03-14 09:44:27.239501 | controller |     - '        fi'
2026-03-14 09:44:27.239505 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.239510 | controller |     - '        '
2026-03-14 09:44:27.239514 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.239518 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.239522 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.239526 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.239530 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.239535 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.239539 | controller |       -- \'
2026-03-14 09:44:27.239543 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.239547 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.239552 | controller |     - '    logger.go:42: 09:26:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.239556 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.239560 | controller |     - '    logger.go:42: 09:26:15 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.239564 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.239568 | controller |       found'
2026-03-14 09:44:27.239579 | controller |     - '    logger.go:42: 09:26:16 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.239584 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.239588 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.239593 | controller |     - '        '
2026-03-14 09:44:27.239597 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.239601 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.239605 | controller |     - '        '
2026-03-14 09:44:27.239609 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.239614 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.239618 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.239622 | controller |     - '          exit 1'
2026-03-14 09:44:27.239626 | controller |     - '        fi'
2026-03-14 09:44:27.239630 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.239635 | controller |     - '        '
2026-03-14 09:44:27.239639 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.239643 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.239647 | controller |       | base64 -d)'
2026-03-14 09:44:27.239651 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.239655 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.239660 | controller |     - '          exit 1'
2026-03-14 09:44:27.239664 | controller |     - '        fi'
2026-03-14 09:44:27.239668 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.239672 | controller |     - '        '
2026-03-14 09:44:27.239679 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.239683 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.239688 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.239692 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.239696 | controller |       --timeout=300s'
2026-03-14 09:44:27.239702 | controller |     - '        '
2026-03-14 09:44:27.239706 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.239710 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.239714 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.239718 | controller |     - '        '
2026-03-14 09:44:27.239723 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.239727 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.239731 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.239735 | controller |     - '        '
2026-03-14 09:44:27.239739 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.239744 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.239748 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.239752 | controller |       available"'
2026-03-14 09:44:27.239756 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.239760 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.239764 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.239769 | controller |     - '          exit 1'
2026-03-14 09:44:27.239773 | controller |     - '        fi'
2026-03-14 09:44:27.239777 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.239781 | controller |     - '        '
2026-03-14 09:44:27.239785 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.239789 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.239794 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.239798 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.239802 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.239806 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.239810 | controller |       -- \'
2026-03-14 09:44:27.239815 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.239820 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.239824 | controller |     - '    logger.go:42: 09:26:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.239829 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.239833 | controller |     - '    logger.go:42: 09:26:16 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.239837 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.239841 | controller |       found'
2026-03-14 09:44:27.239845 | controller |     - '    logger.go:42: 09:26:17 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.239851 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.239861 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.239866 | controller |     - '        '
2026-03-14 09:44:27.239870 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.239874 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.239879 | controller |     - '        '
2026-03-14 09:44:27.239883 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.239887 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.239891 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.239895 | controller |     - '          exit 1'
2026-03-14 09:44:27.239900 | controller |     - '        fi'
2026-03-14 09:44:27.239904 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.239908 | controller |     - '        '
2026-03-14 09:44:27.239912 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.239916 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.239921 | controller |       | base64 -d)'
2026-03-14 09:44:27.239925 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.239929 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.239933 | controller |     - '          exit 1'
2026-03-14 09:44:27.239937 | controller |     - '        fi'
2026-03-14 09:44:27.239942 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.239946 | controller |     - '        '
2026-03-14 09:44:27.239950 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.239954 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.239960 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.239964 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.239968 | controller |       --timeout=300s'
2026-03-14 09:44:27.239973 | controller |     - '        '
2026-03-14 09:44:27.239977 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.239981 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.239986 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.239991 | controller |     - '        '
2026-03-14 09:44:27.239995 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.239999 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.240003 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.240007 | controller |     - '        '
2026-03-14 09:44:27.240012 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.240016 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.240020 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.240024 | controller |       available"'
2026-03-14 09:44:27.240028 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.240033 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.240037 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.240041 | controller |     - '          exit 1'
2026-03-14 09:44:27.240045 | controller |     - '        fi'
2026-03-14 09:44:27.240049 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.240055 | controller |     - '        '
2026-03-14 09:44:27.240059 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.240063 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.240068 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.240072 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.240076 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.240080 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.240084 | controller |       -- \'
2026-03-14 09:44:27.240089 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.240093 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.240097 | controller |     - '    logger.go:42: 09:26:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.240101 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.240105 | controller |     - '    logger.go:42: 09:26:18 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.240110 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.240114 | controller |       found'
2026-03-14 09:44:27.240118 | controller |     - '    logger.go:42: 09:26:19 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.240122 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.240126 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.240131 | controller |     - '        '
2026-03-14 09:44:27.240141 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.240145 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.240150 | controller |     - '        '
2026-03-14 09:44:27.240154 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.240158 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.240164 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.240168 | controller |     - '          exit 1'
2026-03-14 09:44:27.240173 | controller |     - '        fi'
2026-03-14 09:44:27.240177 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.240181 | controller |     - '        '
2026-03-14 09:44:27.240185 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.240190 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.240194 | controller |       | base64 -d)'
2026-03-14 09:44:27.240198 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.240202 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.240207 | controller |     - '          exit 1'
2026-03-14 09:44:27.240211 | controller |     - '        fi'
2026-03-14 09:44:27.240215 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.240219 | controller |     - '        '
2026-03-14 09:44:27.240224 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.240228 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.240232 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.240236 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.240240 | controller |       --timeout=300s'
2026-03-14 09:44:27.240245 | controller |     - '        '
2026-03-14 09:44:27.240249 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.240255 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.240259 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.240263 | controller |     - '        '
2026-03-14 09:44:27.240267 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.240272 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.240276 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.240280 | controller |     - '        '
2026-03-14 09:44:27.240284 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.240301 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.240308 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.240313 | controller |       available"'
2026-03-14 09:44:27.240317 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.240321 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.240326 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.240330 | controller |     - '          exit 1'
2026-03-14 09:44:27.240334 | controller |     - '        fi'
2026-03-14 09:44:27.240338 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.240343 | controller |     - '        '
2026-03-14 09:44:27.240347 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.240351 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.240355 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.240360 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.240364 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.240368 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.240372 | controller |       -- \'
2026-03-14 09:44:27.240377 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.240381 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.240385 | controller |     - '    logger.go:42: 09:26:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.240389 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.240394 | controller |     - '    logger.go:42: 09:26:19 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.240398 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.240402 | controller |       found'
2026-03-14 09:44:27.240408 | controller |     - '    logger.go:42: 09:26:20 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.240414 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.240420 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.240425 | controller |     - '        '
2026-03-14 09:44:27.240439 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.240444 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.240448 | controller |     - '        '
2026-03-14 09:44:27.240452 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.240460 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.240464 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.240468 | controller |     - '          exit 1'
2026-03-14 09:44:27.240472 | controller |     - '        fi'
2026-03-14 09:44:27.240477 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.240481 | controller |     - '        '
2026-03-14 09:44:27.240485 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.240490 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.240496 | controller |       | base64 -d)'
2026-03-14 09:44:27.240502 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.240506 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.240510 | controller |     - '          exit 1'
2026-03-14 09:44:27.240515 | controller |     - '        fi'
2026-03-14 09:44:27.240519 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.240523 | controller |     - '        '
2026-03-14 09:44:27.240527 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.240532 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.240536 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.240542 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.240546 | controller |       --timeout=300s'
2026-03-14 09:44:27.240550 | controller |     - '        '
2026-03-14 09:44:27.240555 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.240559 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.240563 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.240567 | controller |     - '        '
2026-03-14 09:44:27.240573 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.240579 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.240584 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.240588 | controller |     - '        '
2026-03-14 09:44:27.240592 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.240596 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.240601 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.240605 | controller |       available"'
2026-03-14 09:44:27.240609 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.240615 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.240619 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.240623 | controller |     - '          exit 1'
2026-03-14 09:44:27.240628 | controller |     - '        fi'
2026-03-14 09:44:27.240632 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.240636 | controller |     - '        '
2026-03-14 09:44:27.240640 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.240644 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.240650 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.240655 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.240662 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.240666 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.240670 | controller |       -- \'
2026-03-14 09:44:27.240675 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.240679 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.240685 | controller |     - '    logger.go:42: 09:26:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.240689 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.240693 | controller |     - '    logger.go:42: 09:26:20 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.240697 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.240702 | controller |       found'
2026-03-14 09:44:27.240706 | controller |     - '    logger.go:42: 09:26:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.240710 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.240714 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.240719 | controller |     - '        '
2026-03-14 09:44:27.240724 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.240736 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.240743 | controller |     - '        '
2026-03-14 09:44:27.240748 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.240754 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.240760 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.240766 | controller |     - '          exit 1'
2026-03-14 09:44:27.240786 | controller |     - '        fi'
2026-03-14 09:44:27.240791 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.240795 | controller |     - '        '
2026-03-14 09:44:27.240799 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.240804 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.240810 | controller |       | base64 -d)'
2026-03-14 09:44:27.240816 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.240820 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.240824 | controller |     - '          exit 1'
2026-03-14 09:44:27.240829 | controller |     - '        fi'
2026-03-14 09:44:27.240833 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.240837 | controller |     - '        '
2026-03-14 09:44:27.240841 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.240845 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.240849 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.240854 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.240858 | controller |       --timeout=300s'
2026-03-14 09:44:27.240862 | controller |     - '        '
2026-03-14 09:44:27.240866 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.240870 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.240875 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.240879 | controller |     - '        '
2026-03-14 09:44:27.240884 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.240889 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.240898 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.240902 | controller |     - '        '
2026-03-14 09:44:27.240906 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.240911 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.240915 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.240920 | controller |       available"'
2026-03-14 09:44:27.240925 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.240930 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.240934 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.240940 | controller |     - '          exit 1'
2026-03-14 09:44:27.240945 | controller |     - '        fi'
2026-03-14 09:44:27.240949 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.240953 | controller |     - '        '
2026-03-14 09:44:27.240958 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.240963 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.240969 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.240974 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.240981 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.240986 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.240990 | controller |       -- \'
2026-03-14 09:44:27.240994 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.240998 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.241002 | controller |     - '    logger.go:42: 09:26:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.241007 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.241011 | controller |     - '    logger.go:42: 09:26:21 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.241015 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.241019 | controller |       found'
2026-03-14 09:44:27.241023 | controller |     - '    logger.go:42: 09:26:22 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.241028 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.241032 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.241036 | controller |     - '        '
2026-03-14 09:44:27.241041 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.241055 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.241060 | controller |     - '        '
2026-03-14 09:44:27.241065 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.241069 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.241073 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.241077 | controller |     - '          exit 1'
2026-03-14 09:44:27.241082 | controller |     - '        fi'
2026-03-14 09:44:27.241086 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.241090 | controller |     - '        '
2026-03-14 09:44:27.241094 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.241099 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.241105 | controller |       | base64 -d)'
2026-03-14 09:44:27.241109 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.241113 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.241119 | controller |     - '          exit 1'
2026-03-14 09:44:27.241124 | controller |     - '        fi'
2026-03-14 09:44:27.241130 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.241136 | controller |     - '        '
2026-03-14 09:44:27.241142 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.241147 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.241152 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.241156 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.241161 | controller |       --timeout=300s'
2026-03-14 09:44:27.241165 | controller |     - '        '
2026-03-14 09:44:27.241169 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.241173 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.241177 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.241181 | controller |     - '        '
2026-03-14 09:44:27.241186 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.241190 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.241194 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.241200 | controller |     - '        '
2026-03-14 09:44:27.241208 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.241213 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.241217 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.241221 | controller |       available"'
2026-03-14 09:44:27.241225 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.241229 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.241234 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.241238 | controller |     - '          exit 1'
2026-03-14 09:44:27.241242 | controller |     - '        fi'
2026-03-14 09:44:27.241246 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.241250 | controller |     - '        '
2026-03-14 09:44:27.241254 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.241259 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.241263 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.241267 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.241271 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.241276 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.241282 | controller |       -- \'
2026-03-14 09:44:27.241302 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.241311 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.241316 | controller |     - '    logger.go:42: 09:26:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.241322 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.241326 | controller |     - '    logger.go:42: 09:26:22 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.241330 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.241335 | controller |       found'
2026-03-14 09:44:27.241339 | controller |     - '    logger.go:42: 09:26:23 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.241343 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.241347 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.241352 | controller |     - '        '
2026-03-14 09:44:27.241357 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.241362 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.241368 | controller |     - '        '
2026-03-14 09:44:27.241380 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.241385 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.241389 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.241393 | controller |     - '          exit 1'
2026-03-14 09:44:27.241397 | controller |     - '        fi'
2026-03-14 09:44:27.241402 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.241406 | controller |     - '        '
2026-03-14 09:44:27.241410 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.241414 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.241418 | controller |       | base64 -d)'
2026-03-14 09:44:27.241423 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.241427 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.241431 | controller |     - '          exit 1'
2026-03-14 09:44:27.241436 | controller |     - '        fi'
2026-03-14 09:44:27.241442 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.241447 | controller |     - '        '
2026-03-14 09:44:27.241451 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.241456 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.241460 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.241464 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.241468 | controller |       --timeout=300s'
2026-03-14 09:44:27.241472 | controller |     - '        '
2026-03-14 09:44:27.241477 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.241481 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.241485 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.241489 | controller |     - '        '
2026-03-14 09:44:27.241493 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.241497 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.241502 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.241506 | controller |     - '        '
2026-03-14 09:44:27.241513 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.241519 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.241524 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.241530 | controller |       available"'
2026-03-14 09:44:27.241535 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.241539 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.241543 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.241547 | controller |     - '          exit 1'
2026-03-14 09:44:27.241551 | controller |     - '        fi'
2026-03-14 09:44:27.241556 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.241560 | controller |     - '        '
2026-03-14 09:44:27.241564 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.241568 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.241572 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.241577 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.241581 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.241585 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.241589 | controller |       -- \'
2026-03-14 09:44:27.241595 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.241601 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.241606 | controller |     - '    logger.go:42: 09:26:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.241610 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.241614 | controller |     - '    logger.go:42: 09:26:23 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.241618 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.241623 | controller |       found'
2026-03-14 09:44:27.241627 | controller |     - '    logger.go:42: 09:26:24 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.241631 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.241635 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.241639 | controller |     - '        '
2026-03-14 09:44:27.241644 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.241648 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.241652 | controller |     - '        '
2026-03-14 09:44:27.241662 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.241667 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.241672 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.241678 | controller |     - '          exit 1'
2026-03-14 09:44:27.241683 | controller |     - '        fi'
2026-03-14 09:44:27.241687 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.241692 | controller |     - '        '
2026-03-14 09:44:27.241696 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.241700 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.241704 | controller |       | base64 -d)'
2026-03-14 09:44:27.241708 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.241712 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.241717 | controller |     - '          exit 1'
2026-03-14 09:44:27.241721 | controller |     - '        fi'
2026-03-14 09:44:27.241725 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.241729 | controller |     - '        '
2026-03-14 09:44:27.241733 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.241739 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.241744 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.241749 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.241754 | controller |       --timeout=300s'
2026-03-14 09:44:27.241760 | controller |     - '        '
2026-03-14 09:44:27.241764 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.241769 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.241813 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.241818 | controller |     - '        '
2026-03-14 09:44:27.241822 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.241827 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.241832 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.241838 | controller |     - '        '
2026-03-14 09:44:27.241843 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.241847 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.241854 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.241859 | controller |       available"'
2026-03-14 09:44:27.241863 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.241867 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.241872 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.241876 | controller |     - '          exit 1'
2026-03-14 09:44:27.241880 | controller |     - '        fi'
2026-03-14 09:44:27.241884 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.241888 | controller |     - '        '
2026-03-14 09:44:27.241893 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.241897 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.241901 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.241906 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.241912 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.241919 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.241926 | controller |       -- \'
2026-03-14 09:44:27.241933 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.241940 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.241946 | controller |     - '    logger.go:42: 09:26:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.241951 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.241957 | controller |     - '    logger.go:42: 09:26:24 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.241963 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.241969 | controller |       found'
2026-03-14 09:44:27.241974 | controller |     - '    logger.go:42: 09:26:25 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.241979 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.241985 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.241990 | controller |     - '        '
2026-03-14 09:44:27.241994 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.241998 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.242002 | controller |     - '        '
2026-03-14 09:44:27.242006 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.242011 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.242024 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.242030 | controller |     - '          exit 1'
2026-03-14 09:44:27.242036 | controller |     - '        fi'
2026-03-14 09:44:27.242043 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.242049 | controller |     - '        '
2026-03-14 09:44:27.242059 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.242065 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.242071 | controller |       | base64 -d)'
2026-03-14 09:44:27.242076 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.242080 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.242084 | controller |     - '          exit 1'
2026-03-14 09:44:27.242088 | controller |     - '        fi'
2026-03-14 09:44:27.242093 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.242097 | controller |     - '        '
2026-03-14 09:44:27.242101 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.242105 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.242109 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.242114 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.242120 | controller |       --timeout=300s'
2026-03-14 09:44:27.242126 | controller |     - '        '
2026-03-14 09:44:27.242132 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.242138 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.242144 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.242149 | controller |     - '        '
2026-03-14 09:44:27.242162 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.242171 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.242175 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.242179 | controller |     - '        '
2026-03-14 09:44:27.242184 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.242188 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.242192 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.242196 | controller |       available"'
2026-03-14 09:44:27.242200 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.242206 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.242212 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.242218 | controller |     - '          exit 1'
2026-03-14 09:44:27.242224 | controller |     - '        fi'
2026-03-14 09:44:27.242230 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.242239 | controller |     - '        '
2026-03-14 09:44:27.242245 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.242251 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.242257 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.242262 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.242268 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.242274 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.242280 | controller |       -- \'
2026-03-14 09:44:27.242285 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.242303 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.242314 | controller |     - '    logger.go:42: 09:26:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.242320 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.242326 | controller |     - '    logger.go:42: 09:26:26 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.242331 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.242337 | controller |       found'
2026-03-14 09:44:27.242342 | controller |     - '    logger.go:42: 09:26:27 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.242348 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.242353 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.242359 | controller |     - '        '
2026-03-14 09:44:27.242365 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.242370 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.242376 | controller |     - '        '
2026-03-14 09:44:27.242381 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.242387 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.242402 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.242413 | controller |     - '          exit 1'
2026-03-14 09:44:27.242419 | controller |     - '        fi'
2026-03-14 09:44:27.242425 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.242431 | controller |     - '        '
2026-03-14 09:44:27.242437 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.242443 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.242448 | controller |       | base64 -d)'
2026-03-14 09:44:27.242454 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.242459 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.242464 | controller |     - '          exit 1'
2026-03-14 09:44:27.242470 | controller |     - '        fi'
2026-03-14 09:44:27.242476 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.242482 | controller |     - '        '
2026-03-14 09:44:27.242488 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.242494 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.242500 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.242505 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.242511 | controller |       --timeout=300s'
2026-03-14 09:44:27.242517 | controller |     - '        '
2026-03-14 09:44:27.242523 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.242533 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.242538 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.242544 | controller |     - '        '
2026-03-14 09:44:27.242550 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.242556 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.242562 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.242568 | controller |     - '        '
2026-03-14 09:44:27.242574 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.242579 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.242585 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.242591 | controller |       available"'
2026-03-14 09:44:27.242597 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.242606 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.242612 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.242618 | controller |     - '          exit 1'
2026-03-14 09:44:27.242624 | controller |     - '        fi'
2026-03-14 09:44:27.242629 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.242635 | controller |     - '        '
2026-03-14 09:44:27.242641 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.242647 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.242653 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.242658 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.242664 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.242670 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.242676 | controller |       -- \'
2026-03-14 09:44:27.242682 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.242687 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.242693 | controller |     - '    logger.go:42: 09:26:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.242699 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.242705 | controller |     - '    logger.go:42: 09:26:27 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.242710 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.242716 | controller |       found'
2026-03-14 09:44:27.242722 | controller |     - '    logger.go:42: 09:26:28 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.242728 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.242734 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.242739 | controller |     - '        '
2026-03-14 09:44:27.242745 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.242751 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.242757 | controller |     - '        '
2026-03-14 09:44:27.242763 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.242769 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.242777 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.242783 | controller |     - '          exit 1'
2026-03-14 09:44:27.242798 | controller |     - '        fi'
2026-03-14 09:44:27.242804 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.242810 | controller |     - '        '
2026-03-14 09:44:27.242816 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.242822 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.242827 | controller |       | base64 -d)'
2026-03-14 09:44:27.242833 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.242839 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.242845 | controller |     - '          exit 1'
2026-03-14 09:44:27.242851 | controller |     - '        fi'
2026-03-14 09:44:27.242860 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.242866 | controller |     - '        '
2026-03-14 09:44:27.242872 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.242878 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.242884 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.242889 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.242895 | controller |       --timeout=300s'
2026-03-14 09:44:27.242901 | controller |     - '        '
2026-03-14 09:44:27.242907 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.242913 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.242918 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.242924 | controller |     - '        '
2026-03-14 09:44:27.242929 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.242933 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.242937 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.242941 | controller |     - '        '
2026-03-14 09:44:27.242945 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.242950 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.242954 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.242958 | controller |       available"'
2026-03-14 09:44:27.242962 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.242966 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.242971 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.242975 | controller |     - '          exit 1'
2026-03-14 09:44:27.242979 | controller |     - '        fi'
2026-03-14 09:44:27.242983 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.242987 | controller |     - '        '
2026-03-14 09:44:27.242991 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.242996 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.243000 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.243004 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.243008 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.243014 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.243019 | controller |       -- \'
2026-03-14 09:44:27.243023 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.243027 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.243031 | controller |     - '    logger.go:42: 09:26:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.243035 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.243040 | controller |     - '    logger.go:42: 09:26:28 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.243044 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.243048 | controller |       found'
2026-03-14 09:44:27.243052 | controller |     - '    logger.go:42: 09:26:29 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.243057 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.243061 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.243065 | controller |     - '        '
2026-03-14 09:44:27.243069 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.243074 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.243078 | controller |     - '        '
2026-03-14 09:44:27.243082 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.243086 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.243091 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.243095 | controller |     - '          exit 1'
2026-03-14 09:44:27.243099 | controller |     - '        fi'
2026-03-14 09:44:27.243110 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.243115 | controller |     - '        '
2026-03-14 09:44:27.243120 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.243124 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.243129 | controller |       | base64 -d)'
2026-03-14 09:44:27.243133 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.243137 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.243141 | controller |     - '          exit 1'
2026-03-14 09:44:27.243146 | controller |     - '        fi'
2026-03-14 09:44:27.243150 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.243154 | controller |     - '        '
2026-03-14 09:44:27.243159 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.243163 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.243167 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.243171 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.243175 | controller |       --timeout=300s'
2026-03-14 09:44:27.243180 | controller |     - '        '
2026-03-14 09:44:27.243184 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.243188 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.243193 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.243197 | controller |     - '        '
2026-03-14 09:44:27.243201 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.243205 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.243211 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.243216 | controller |     - '        '
2026-03-14 09:44:27.243220 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.243224 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.243229 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.243233 | controller |       available"'
2026-03-14 09:44:27.243237 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.243242 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.243246 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.243250 | controller |     - '          exit 1'
2026-03-14 09:44:27.243254 | controller |     - '        fi'
2026-03-14 09:44:27.243259 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.243263 | controller |     - '        '
2026-03-14 09:44:27.243269 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.243273 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.243278 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.243282 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.243289 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.243309 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.243319 | controller |       -- \'
2026-03-14 09:44:27.243325 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.243331 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.243337 | controller |     - '    logger.go:42: 09:26:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.243343 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.243349 | controller |     - '    logger.go:42: 09:26:29 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.243354 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.243359 | controller |       found'
2026-03-14 09:44:27.243363 | controller |     - '    logger.go:42: 09:26:30 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.243367 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.243371 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.243376 | controller |     - '        '
2026-03-14 09:44:27.243380 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.243384 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.243388 | controller |     - '        '
2026-03-14 09:44:27.243393 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.243397 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.243401 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.243406 | controller |     - '          exit 1'
2026-03-14 09:44:27.243412 | controller |     - '        fi'
2026-03-14 09:44:27.243418 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.243424 | controller |     - '        '
2026-03-14 09:44:27.243438 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.243445 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.243451 | controller |       | base64 -d)'
2026-03-14 09:44:27.243459 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.243464 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.243468 | controller |     - '          exit 1'
2026-03-14 09:44:27.243472 | controller |     - '        fi'
2026-03-14 09:44:27.243476 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.243481 | controller |     - '        '
2026-03-14 09:44:27.243485 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.243489 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.243493 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.243498 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.243504 | controller |       --timeout=300s'
2026-03-14 09:44:27.243510 | controller |     - '        '
2026-03-14 09:44:27.243516 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.243522 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.243528 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.243534 | controller |     - '        '
2026-03-14 09:44:27.243540 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.243546 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.243551 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.243557 | controller |     - '        '
2026-03-14 09:44:27.243562 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.243567 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.243571 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.243612 | controller |       available"'
2026-03-14 09:44:27.243617 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.243625 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.243629 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.243633 | controller |     - '          exit 1'
2026-03-14 09:44:27.243637 | controller |     - '        fi'
2026-03-14 09:44:27.243641 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.243646 | controller |     - '        '
2026-03-14 09:44:27.243650 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.243654 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.243658 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.243662 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.243667 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.243671 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.243675 | controller |       -- \'
2026-03-14 09:44:27.243688 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.243692 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.243696 | controller |     - '    logger.go:42: 09:26:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.243728 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.243735 | controller |     - '    logger.go:42: 09:26:30 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.243741 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.243746 | controller |       found'
2026-03-14 09:44:27.243751 | controller |     - '    logger.go:42: 09:26:31 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.243757 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.243762 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.243768 | controller |     - '        '
2026-03-14 09:44:27.243774 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.243779 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.243785 | controller |     - '        '
2026-03-14 09:44:27.243790 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.243796 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.243801 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.243807 | controller |     - '          exit 1'
2026-03-14 09:44:27.243813 | controller |     - '        fi'
2026-03-14 09:44:27.243818 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.243824 | controller |     - '        '
2026-03-14 09:44:27.243830 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.243848 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.243855 | controller |       | base64 -d)'
2026-03-14 09:44:27.243861 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.243867 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.243873 | controller |     - '          exit 1'
2026-03-14 09:44:27.243878 | controller |     - '        fi'
2026-03-14 09:44:27.243884 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.243888 | controller |     - '        '
2026-03-14 09:44:27.243892 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.243896 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.243901 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.243905 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.243910 | controller |       --timeout=300s'
2026-03-14 09:44:27.243916 | controller |     - '        '
2026-03-14 09:44:27.243921 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.243927 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.243933 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.243939 | controller |     - '        '
2026-03-14 09:44:27.243945 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.243950 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.243956 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.243961 | controller |     - '        '
2026-03-14 09:44:27.243966 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.243972 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.243978 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.243984 | controller |       available"'
2026-03-14 09:44:27.243992 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.243998 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.244004 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.244010 | controller |     - '          exit 1'
2026-03-14 09:44:27.244015 | controller |     - '        fi'
2026-03-14 09:44:27.244021 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.244027 | controller |     - '        '
2026-03-14 09:44:27.244033 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.244038 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.244044 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.244052 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.244058 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.244064 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.244068 | controller |       -- \'
2026-03-14 09:44:27.244073 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.244077 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.244081 | controller |     - '    logger.go:42: 09:26:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.244085 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.244090 | controller |     - '    logger.go:42: 09:26:31 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.244094 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.244098 | controller |       found'
2026-03-14 09:44:27.244102 | controller |     - '    logger.go:42: 09:26:32 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.244106 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.244110 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.244115 | controller |     - '        '
2026-03-14 09:44:27.244119 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.244123 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.244127 | controller |     - '        '
2026-03-14 09:44:27.244131 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.244135 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.244140 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.244144 | controller |     - '          exit 1'
2026-03-14 09:44:27.244150 | controller |     - '        fi'
2026-03-14 09:44:27.244154 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.244158 | controller |     - '        '
2026-03-14 09:44:27.244163 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.244174 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.244180 | controller |       | base64 -d)'
2026-03-14 09:44:27.244184 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.244188 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.244193 | controller |     - '          exit 1'
2026-03-14 09:44:27.244197 | controller |     - '        fi'
2026-03-14 09:44:27.244201 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.244205 | controller |     - '        '
2026-03-14 09:44:27.244209 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.244214 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.244220 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.244224 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.244228 | controller |       --timeout=300s'
2026-03-14 09:44:27.244234 | controller |     - '        '
2026-03-14 09:44:27.244238 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.244243 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.244247 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.244251 | controller |     - '        '
2026-03-14 09:44:27.244255 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.244259 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.244263 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.244268 | controller |     - '        '
2026-03-14 09:44:27.244272 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.244276 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.244280 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.244284 | controller |       available"'
2026-03-14 09:44:27.244311 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.244316 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.244320 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.244324 | controller |     - '          exit 1'
2026-03-14 09:44:27.244329 | controller |     - '        fi'
2026-03-14 09:44:27.244333 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.244337 | controller |     - '        '
2026-03-14 09:44:27.244341 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.244345 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.244350 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.244354 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.244358 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.244362 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.244366 | controller |       -- \'
2026-03-14 09:44:27.244371 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.244375 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.244379 | controller |     - '    logger.go:42: 09:26:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.244383 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.244387 | controller |     - '    logger.go:42: 09:26:32 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.244392 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.244396 | controller |       found'
2026-03-14 09:44:27.244400 | controller |     - '    logger.go:42: 09:26:33 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.244404 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.244409 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.244415 | controller |     - '        '
2026-03-14 09:44:27.244419 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.244424 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.244428 | controller |     - '        '
2026-03-14 09:44:27.244432 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.244436 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.244441 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.244445 | controller |     - '          exit 1'
2026-03-14 09:44:27.244451 | controller |     - '        fi'
2026-03-14 09:44:27.244455 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.244460 | controller |     - '        '
2026-03-14 09:44:27.244464 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.244468 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.244479 | controller |       | base64 -d)'
2026-03-14 09:44:27.244483 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.244487 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.244492 | controller |     - '          exit 1'
2026-03-14 09:44:27.244496 | controller |     - '        fi'
2026-03-14 09:44:27.244500 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.244504 | controller |     - '        '
2026-03-14 09:44:27.244509 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.244513 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.244517 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.244521 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.244525 | controller |       --timeout=300s'
2026-03-14 09:44:27.244529 | controller |     - '        '
2026-03-14 09:44:27.244533 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.244538 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.244542 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.244546 | controller |     - '        '
2026-03-14 09:44:27.244550 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.244554 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.244560 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.244564 | controller |     - '        '
2026-03-14 09:44:27.244569 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.244573 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.244577 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.244581 | controller |       available"'
2026-03-14 09:44:27.244585 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.244589 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.244594 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.244598 | controller |     - '          exit 1'
2026-03-14 09:44:27.244602 | controller |     - '        fi'
2026-03-14 09:44:27.244606 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.244610 | controller |     - '        '
2026-03-14 09:44:27.244615 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.244620 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.244625 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.244629 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.244633 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.244637 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.244642 | controller |       -- \'
2026-03-14 09:44:27.244646 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.244651 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.244656 | controller |     - '    logger.go:42: 09:26:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.244660 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.244664 | controller |     - '    logger.go:42: 09:26:34 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.244668 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.244672 | controller |       found'
2026-03-14 09:44:27.244677 | controller |     - '    logger.go:42: 09:26:35 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.244681 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.244685 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.244689 | controller |     - '        '
2026-03-14 09:44:27.244693 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.244697 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.244702 | controller |     - '        '
2026-03-14 09:44:27.244706 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.244710 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.244714 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.244718 | controller |     - '          exit 1'
2026-03-14 09:44:27.244722 | controller |     - '        fi'
2026-03-14 09:44:27.244727 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.244731 | controller |     - '        '
2026-03-14 09:44:27.244735 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.244739 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.244743 | controller |       | base64 -d)'
2026-03-14 09:44:27.244753 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.244758 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.244762 | controller |     - '          exit 1'
2026-03-14 09:44:27.244766 | controller |     - '        fi'
2026-03-14 09:44:27.244771 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.244775 | controller |     - '        '
2026-03-14 09:44:27.244779 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.244783 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.244787 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.244792 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.244796 | controller |       --timeout=300s'
2026-03-14 09:44:27.244800 | controller |     - '        '
2026-03-14 09:44:27.244804 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.244808 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.244814 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.244819 | controller |     - '        '
2026-03-14 09:44:27.244823 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.244827 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.244831 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.244835 | controller |     - '        '
2026-03-14 09:44:27.244839 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.244844 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.244848 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.244852 | controller |       available"'
2026-03-14 09:44:27.244856 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.244860 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.244864 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.244869 | controller |     - '          exit 1'
2026-03-14 09:44:27.244873 | controller |     - '        fi'
2026-03-14 09:44:27.244877 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.244881 | controller |     - '        '
2026-03-14 09:44:27.244885 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.244889 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.244893 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.244898 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.244902 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.244906 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.244911 | controller |       -- \'
2026-03-14 09:44:27.244917 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.244921 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.244926 | controller |     - '    logger.go:42: 09:26:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.244931 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.244936 | controller |     - '    logger.go:42: 09:26:35 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.244941 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.244945 | controller |       found'
2026-03-14 09:44:27.244949 | controller |     - '    logger.go:42: 09:26:36 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.244953 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.244958 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.244962 | controller |     - '        '
2026-03-14 09:44:27.244966 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.244970 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.244974 | controller |     - '        '
2026-03-14 09:44:27.244978 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.244983 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.244987 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.244992 | controller |     - '          exit 1'
2026-03-14 09:44:27.244997 | controller |     - '        fi'
2026-03-14 09:44:27.245001 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.245005 | controller |     - '        '
2026-03-14 09:44:27.245024 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.245028 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.245032 | controller |       | base64 -d)'
2026-03-14 09:44:27.245037 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.245047 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.245051 | controller |     - '          exit 1'
2026-03-14 09:44:27.245055 | controller |     - '        fi'
2026-03-14 09:44:27.245060 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.245064 | controller |     - '        '
2026-03-14 09:44:27.245068 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.245072 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.245076 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.245083 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.245088 | controller |       --timeout=300s'
2026-03-14 09:44:27.245093 | controller |     - '        '
2026-03-14 09:44:27.245097 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.245101 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.245105 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.245110 | controller |     - '        '
2026-03-14 09:44:27.245114 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.245118 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.245122 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.245126 | controller |     - '        '
2026-03-14 09:44:27.245131 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.245135 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.245139 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.245143 | controller |       available"'
2026-03-14 09:44:27.245147 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.245151 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.245156 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.245160 | controller |     - '          exit 1'
2026-03-14 09:44:27.245167 | controller |     - '        fi'
2026-03-14 09:44:27.245171 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.245175 | controller |     - '        '
2026-03-14 09:44:27.245179 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.245183 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.245188 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.245192 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.245196 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.245202 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.245206 | controller |       -- \'
2026-03-14 09:44:27.245210 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.245214 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.245220 | controller |     - '    logger.go:42: 09:26:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.245224 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.245230 | controller |     - '    logger.go:42: 09:26:36 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.245234 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.245238 | controller |       found'
2026-03-14 09:44:27.245242 | controller |     - '    logger.go:42: 09:26:37 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.245247 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.245251 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.245255 | controller |     - '        '
2026-03-14 09:44:27.245259 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.245263 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.245268 | controller |     - '        '
2026-03-14 09:44:27.245272 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.245276 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.245280 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.245284 | controller |     - '          exit 1'
2026-03-14 09:44:27.245302 | controller |     - '        fi'
2026-03-14 09:44:27.245309 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.245314 | controller |     - '        '
2026-03-14 09:44:27.245318 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.245322 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.245326 | controller |       | base64 -d)'
2026-03-14 09:44:27.245330 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.245341 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.245345 | controller |     - '          exit 1'
2026-03-14 09:44:27.245349 | controller |     - '        fi'
2026-03-14 09:44:27.245353 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.245358 | controller |     - '        '
2026-03-14 09:44:27.245362 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.245366 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.245370 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.245374 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.245379 | controller |       --timeout=300s'
2026-03-14 09:44:27.245383 | controller |     - '        '
2026-03-14 09:44:27.245387 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.245391 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.245395 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.245399 | controller |     - '        '
2026-03-14 09:44:27.245404 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.245408 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.245412 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.245418 | controller |     - '        '
2026-03-14 09:44:27.245422 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.245426 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.245430 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.245435 | controller |       available"'
2026-03-14 09:44:27.245439 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.245443 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.245447 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.245451 | controller |     - '          exit 1'
2026-03-14 09:44:27.245455 | controller |     - '        fi'
2026-03-14 09:44:27.245459 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.245464 | controller |     - '        '
2026-03-14 09:44:27.245468 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.245472 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.245476 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.245480 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.245484 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.245489 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.245493 | controller |       -- \'
2026-03-14 09:44:27.245497 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.245501 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.245507 | controller |     - '    logger.go:42: 09:26:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.245511 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.245516 | controller |     - '    logger.go:42: 09:26:37 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.245520 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.245524 | controller |       found'
2026-03-14 09:44:27.245528 | controller |     - '    logger.go:42: 09:26:38 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.245532 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.245537 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.245541 | controller |     - '        '
2026-03-14 09:44:27.245545 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.245549 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.245553 | controller |     - '        '
2026-03-14 09:44:27.245558 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.245562 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.245566 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.245570 | controller |     - '          exit 1'
2026-03-14 09:44:27.245574 | controller |     - '        fi'
2026-03-14 09:44:27.245578 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.245583 | controller |     - '        '
2026-03-14 09:44:27.245587 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.245591 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.245595 | controller |       | base64 -d)'
2026-03-14 09:44:27.245599 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.245605 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.245609 | controller |     - '          exit 1'
2026-03-14 09:44:27.245613 | controller |     - '        fi'
2026-03-14 09:44:27.245623 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.245628 | controller |     - '        '
2026-03-14 09:44:27.245632 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.245636 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.245640 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.245645 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.245649 | controller |       --timeout=300s'
2026-03-14 09:44:27.245653 | controller |     - '        '
2026-03-14 09:44:27.245657 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.245661 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.245666 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.245670 | controller |     - '        '
2026-03-14 09:44:27.245674 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.245678 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.245683 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.245687 | controller |     - '        '
2026-03-14 09:44:27.245692 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.245696 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.245700 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.245704 | controller |       available"'
2026-03-14 09:44:27.245708 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.245713 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.245717 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.245721 | controller |     - '          exit 1'
2026-03-14 09:44:27.245725 | controller |     - '        fi'
2026-03-14 09:44:27.245729 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.245734 | controller |     - '        '
2026-03-14 09:44:27.245738 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.245742 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.245746 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.245750 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.245754 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.245759 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.245763 | controller |       -- \'
2026-03-14 09:44:27.245767 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.245771 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.245775 | controller |     - '    logger.go:42: 09:26:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.245797 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.245806 | controller |     - '    logger.go:42: 09:26:38 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.245814 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.245819 | controller |       found'
2026-03-14 09:44:27.245824 | controller |     - '    logger.go:42: 09:26:39 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.245829 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.245834 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.245839 | controller |     - '        '
2026-03-14 09:44:27.245844 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.245849 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.245854 | controller |     - '        '
2026-03-14 09:44:27.245859 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.245864 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.245871 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.245877 | controller |     - '          exit 1'
2026-03-14 09:44:27.245882 | controller |     - '        fi'
2026-03-14 09:44:27.245888 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.245894 | controller |     - '        '
2026-03-14 09:44:27.245900 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.245905 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.245911 | controller |       | base64 -d)'
2026-03-14 09:44:27.245917 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.245923 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.245928 | controller |     - '          exit 1'
2026-03-14 09:44:27.245934 | controller |     - '        fi'
2026-03-14 09:44:27.245948 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.245954 | controller |     - '        '
2026-03-14 09:44:27.245960 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.245966 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.245971 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.245977 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.245982 | controller |       --timeout=300s'
2026-03-14 09:44:27.245988 | controller |     - '        '
2026-03-14 09:44:27.245993 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.245999 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.246004 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.246009 | controller |     - '        '
2026-03-14 09:44:27.246014 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.246020 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.246026 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.246031 | controller |     - '        '
2026-03-14 09:44:27.246037 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.246042 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.246048 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.246054 | controller |       available"'
2026-03-14 09:44:27.246060 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.246069 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.246075 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.246081 | controller |     - '          exit 1'
2026-03-14 09:44:27.246087 | controller |     - '        fi'
2026-03-14 09:44:27.246093 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.246099 | controller |     - '        '
2026-03-14 09:44:27.246105 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.246110 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.246116 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.246122 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.246128 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.246133 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.246139 | controller |       -- \'
2026-03-14 09:44:27.246145 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.246150 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.246156 | controller |     - '    logger.go:42: 09:26:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.246162 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.246167 | controller |     - '    logger.go:42: 09:26:40 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.246173 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.246179 | controller |       found'
2026-03-14 09:44:27.246184 | controller |     - '    logger.go:42: 09:26:41 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.246190 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.246196 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.246201 | controller |     - '        '
2026-03-14 09:44:27.246207 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.246212 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.246218 | controller |     - '        '
2026-03-14 09:44:27.246223 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.246229 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.246235 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.246240 | controller |     - '          exit 1'
2026-03-14 09:44:27.246246 | controller |     - '        fi'
2026-03-14 09:44:27.246251 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.246257 | controller |     - '        '
2026-03-14 09:44:27.246262 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.246268 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.246274 | controller |       | base64 -d)'
2026-03-14 09:44:27.246280 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.246286 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.246313 | controller |     - '          exit 1'
2026-03-14 09:44:27.246319 | controller |     - '        fi'
2026-03-14 09:44:27.246325 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.246331 | controller |     - '        '
2026-03-14 09:44:27.246345 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.246352 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.246360 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.246366 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.246372 | controller |       --timeout=300s'
2026-03-14 09:44:27.246382 | controller |     - '        '
2026-03-14 09:44:27.246387 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.246394 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.246400 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.246405 | controller |     - '        '
2026-03-14 09:44:27.246411 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.246417 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.246423 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.246429 | controller |     - '        '
2026-03-14 09:44:27.246435 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.246441 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.246447 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.246452 | controller |       available"'
2026-03-14 09:44:27.246458 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.246466 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.246472 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.246478 | controller |     - '          exit 1'
2026-03-14 09:44:27.246484 | controller |     - '        fi'
2026-03-14 09:44:27.246490 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.246496 | controller |     - '        '
2026-03-14 09:44:27.246501 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.246507 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.246513 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.246519 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.246525 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.246531 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.246537 | controller |       -- \'
2026-03-14 09:44:27.246542 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.246548 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.246554 | controller |     - '    logger.go:42: 09:26:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.246560 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.246566 | controller |     - '    logger.go:42: 09:26:41 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.246572 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.246577 | controller |       found'
2026-03-14 09:44:27.246585 | controller |     - '    logger.go:42: 09:26:42 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.246591 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.246597 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.246603 | controller |     - '        '
2026-03-14 09:44:27.246612 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.246618 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.246624 | controller |     - '        '
2026-03-14 09:44:27.246630 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.246636 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.246642 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.246648 | controller |     - '          exit 1'
2026-03-14 09:44:27.246654 | controller |     - '        fi'
2026-03-14 09:44:27.246659 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.246665 | controller |     - '        '
2026-03-14 09:44:27.246671 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.246677 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.246683 | controller |       | base64 -d)'
2026-03-14 09:44:27.246689 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.246695 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.246700 | controller |     - '          exit 1'
2026-03-14 09:44:27.246706 | controller |     - '        fi'
2026-03-14 09:44:27.246712 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.246718 | controller |     - '        '
2026-03-14 09:44:27.246724 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.246738 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.246744 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.246750 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.246756 | controller |       --timeout=300s'
2026-03-14 09:44:27.246762 | controller |     - '        '
2026-03-14 09:44:27.246768 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.246773 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.246779 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.246785 | controller |     - '        '
2026-03-14 09:44:27.246794 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.246800 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.246806 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.246811 | controller |     - '        '
2026-03-14 09:44:27.246817 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.246824 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.246830 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.246836 | controller |       available"'
2026-03-14 09:44:27.246842 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.246847 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.246853 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.246859 | controller |     - '          exit 1'
2026-03-14 09:44:27.246865 | controller |     - '        fi'
2026-03-14 09:44:27.246871 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.246877 | controller |     - '        '
2026-03-14 09:44:27.246883 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.246892 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.246898 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.246904 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.246911 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.246918 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.246925 | controller |       -- \'
2026-03-14 09:44:27.246932 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.246939 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.246945 | controller |     - '    logger.go:42: 09:26:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.246951 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.246957 | controller |     - '    logger.go:42: 09:26:42 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.246963 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.246969 | controller |       found'
2026-03-14 09:44:27.246975 | controller |     - '    logger.go:42: 09:26:43 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.246981 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.246987 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.246993 | controller |     - '        '
2026-03-14 09:44:27.246999 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.247005 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.247011 | controller |     - '        '
2026-03-14 09:44:27.247017 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.247023 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.247029 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.247035 | controller |     - '          exit 1'
2026-03-14 09:44:27.247041 | controller |     - '        fi'
2026-03-14 09:44:27.247047 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.247052 | controller |     - '        '
2026-03-14 09:44:27.247058 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.247064 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.247070 | controller |       | base64 -d)'
2026-03-14 09:44:27.247076 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.247083 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.247088 | controller |     - '          exit 1'
2026-03-14 09:44:27.247094 | controller |     - '        fi'
2026-03-14 09:44:27.247100 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.247106 | controller |     - '        '
2026-03-14 09:44:27.247112 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.247126 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.247133 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.247139 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.247144 | controller |       --timeout=300s'
2026-03-14 09:44:27.247150 | controller |     - '        '
2026-03-14 09:44:27.247156 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.247162 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.247171 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.247177 | controller |     - '        '
2026-03-14 09:44:27.247183 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.247189 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.247195 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.247201 | controller |     - '        '
2026-03-14 09:44:27.247207 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.247212 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.247218 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.247224 | controller |       available"'
2026-03-14 09:44:27.247229 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.247235 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.247241 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.247247 | controller |     - '          exit 1'
2026-03-14 09:44:27.247253 | controller |     - '        fi'
2026-03-14 09:44:27.247259 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.247265 | controller |     - '        '
2026-03-14 09:44:27.247271 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.247276 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.247282 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.247338 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.247353 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.247359 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.247365 | controller |       -- \'
2026-03-14 09:44:27.247370 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.247376 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.247382 | controller |     - '    logger.go:42: 09:26:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.247388 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.247393 | controller |     - '    logger.go:42: 09:26:43 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.247399 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.247405 | controller |       found'
2026-03-14 09:44:27.247411 | controller |     - '    logger.go:42: 09:26:44 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.247416 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.247422 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.247428 | controller |     - '        '
2026-03-14 09:44:27.247437 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.247444 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.247449 | controller |     - '        '
2026-03-14 09:44:27.247455 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.247510 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.247516 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.247520 | controller |     - '          exit 1'
2026-03-14 09:44:27.247527 | controller |     - '        fi'
2026-03-14 09:44:27.247531 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.247535 | controller |     - '        '
2026-03-14 09:44:27.247539 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.247544 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.247548 | controller |       | base64 -d)'
2026-03-14 09:44:27.247552 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.247556 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.247561 | controller |     - '          exit 1'
2026-03-14 09:44:27.247565 | controller |     - '        fi'
2026-03-14 09:44:27.247569 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.247573 | controller |     - '        '
2026-03-14 09:44:27.247577 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.247582 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.247594 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.247599 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.247603 | controller |       --timeout=300s'
2026-03-14 09:44:27.247607 | controller |     - '        '
2026-03-14 09:44:27.247611 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.247617 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.247623 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.247629 | controller |     - '        '
2026-03-14 09:44:27.247635 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.247641 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.247647 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.247653 | controller |     - '        '
2026-03-14 09:44:27.247658 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.247662 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.247667 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.247671 | controller |       available"'
2026-03-14 09:44:27.247675 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.247679 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.247683 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.247688 | controller |     - '          exit 1'
2026-03-14 09:44:27.247692 | controller |     - '        fi'
2026-03-14 09:44:27.247696 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.247700 | controller |     - '        '
2026-03-14 09:44:27.247704 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.247708 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.247713 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.247717 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.247721 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.247725 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.247732 | controller |       -- \'
2026-03-14 09:44:27.247736 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.247740 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.247744 | controller |     - '    logger.go:42: 09:26:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.247748 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.247753 | controller |     - '    logger.go:42: 09:26:44 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.247757 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.247761 | controller |       found'
2026-03-14 09:44:27.247765 | controller |     - '    logger.go:42: 09:26:45 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.247769 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.247773 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.247778 | controller |     - '        '
2026-03-14 09:44:27.247782 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.247786 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.247790 | controller |     - '        '
2026-03-14 09:44:27.247794 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.247799 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.247803 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.247807 | controller |     - '          exit 1'
2026-03-14 09:44:27.247811 | controller |     - '        fi'
2026-03-14 09:44:27.247815 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.247820 | controller |     - '        '
2026-03-14 09:44:27.247827 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.247831 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.247835 | controller |       | base64 -d)'
2026-03-14 09:44:27.247840 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.247844 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.247848 | controller |     - '          exit 1'
2026-03-14 09:44:27.247852 | controller |     - '        fi'
2026-03-14 09:44:27.247856 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.247861 | controller |     - '        '
2026-03-14 09:44:27.247865 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.247869 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.247880 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.247885 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.247889 | controller |       --timeout=300s'
2026-03-14 09:44:27.247893 | controller |     - '        '
2026-03-14 09:44:27.247897 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.247902 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.247906 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.247910 | controller |     - '        '
2026-03-14 09:44:27.247915 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.247919 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.247924 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.247930 | controller |     - '        '
2026-03-14 09:44:27.247934 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.247938 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.247942 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.247946 | controller |       available"'
2026-03-14 09:44:27.247951 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.247955 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.247959 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.247963 | controller |     - '          exit 1'
2026-03-14 09:44:27.247967 | controller |     - '        fi'
2026-03-14 09:44:27.247972 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.247976 | controller |     - '        '
2026-03-14 09:44:27.247980 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.247984 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.247988 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.247992 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.247997 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.248001 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.248005 | controller |       -- \'
2026-03-14 09:44:27.248009 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.248013 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.248018 | controller |     - '    logger.go:42: 09:26:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.248022 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.248026 | controller |     - '    logger.go:42: 09:26:45 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.248030 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.248035 | controller |       found'
2026-03-14 09:44:27.248039 | controller |     - '    logger.go:42: 09:26:46 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.248043 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.248047 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.248051 | controller |     - '        '
2026-03-14 09:44:27.248056 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.248061 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.248066 | controller |     - '        '
2026-03-14 09:44:27.248070 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.248074 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.248078 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.248082 | controller |     - '          exit 1'
2026-03-14 09:44:27.248086 | controller |     - '        fi'
2026-03-14 09:44:27.248091 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.248095 | controller |     - '        '
2026-03-14 09:44:27.248100 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.248105 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.248109 | controller |       | base64 -d)'
2026-03-14 09:44:27.248113 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.248117 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.248123 | controller |     - '          exit 1'
2026-03-14 09:44:27.248127 | controller |     - '        fi'
2026-03-14 09:44:27.248131 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.248136 | controller |     - '        '
2026-03-14 09:44:27.248140 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.248144 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.248148 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.248159 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.248163 | controller |       --timeout=300s'
2026-03-14 09:44:27.248168 | controller |     - '        '
2026-03-14 09:44:27.248172 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.248176 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.248182 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.248186 | controller |     - '        '
2026-03-14 09:44:27.248190 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.248194 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.248199 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.248203 | controller |     - '        '
2026-03-14 09:44:27.248207 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.248211 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.248215 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.248219 | controller |       available"'
2026-03-14 09:44:27.248224 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.248228 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.248232 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.248236 | controller |     - '          exit 1'
2026-03-14 09:44:27.248241 | controller |     - '        fi'
2026-03-14 09:44:27.248245 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.248249 | controller |     - '        '
2026-03-14 09:44:27.248253 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.248257 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.248261 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.248265 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.248270 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.248274 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.248278 | controller |       -- \'
2026-03-14 09:44:27.248282 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.248289 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.248309 | controller |     - '    logger.go:42: 09:26:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.248314 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.248318 | controller |     - '    logger.go:42: 09:26:46 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.248325 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.248329 | controller |       found'
2026-03-14 09:44:27.248333 | controller |     - '    logger.go:42: 09:26:47 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.248337 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.248341 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.248346 | controller |     - '        '
2026-03-14 09:44:27.248350 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.248354 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.248358 | controller |     - '        '
2026-03-14 09:44:27.248362 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.248366 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.248374 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.248378 | controller |     - '          exit 1'
2026-03-14 09:44:27.248383 | controller |     - '        fi'
2026-03-14 09:44:27.248387 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.248391 | controller |     - '        '
2026-03-14 09:44:27.248395 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.248400 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.248404 | controller |       | base64 -d)'
2026-03-14 09:44:27.248408 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.248412 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.248418 | controller |     - '          exit 1'
2026-03-14 09:44:27.248422 | controller |     - '        fi'
2026-03-14 09:44:27.248426 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.248431 | controller |     - '        '
2026-03-14 09:44:27.248435 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.248439 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.248443 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.248448 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.248459 | controller |       --timeout=300s'
2026-03-14 09:44:27.248463 | controller |     - '        '
2026-03-14 09:44:27.248468 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.248472 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.248476 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.248480 | controller |     - '        '
2026-03-14 09:44:27.248484 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.248489 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.248493 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.248497 | controller |     - '        '
2026-03-14 09:44:27.248501 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.248505 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.248510 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.248514 | controller |       available"'
2026-03-14 09:44:27.248518 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.248522 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.248528 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.248532 | controller |     - '          exit 1'
2026-03-14 09:44:27.248537 | controller |     - '        fi'
2026-03-14 09:44:27.248541 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.248545 | controller |     - '        '
2026-03-14 09:44:27.248549 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.248553 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.248558 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.248562 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.248566 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.248570 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.248575 | controller |       -- \'
2026-03-14 09:44:27.248579 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.248583 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.248587 | controller |     - '    logger.go:42: 09:26:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.248592 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.248596 | controller |     - '    logger.go:42: 09:26:48 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.248600 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.248604 | controller |       found'
2026-03-14 09:44:27.248609 | controller |     - '    logger.go:42: 09:26:49 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.248614 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.248619 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.248625 | controller |     - '        '
2026-03-14 09:44:27.248631 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.248637 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.248643 | controller |     - '        '
2026-03-14 09:44:27.248651 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.248657 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.248663 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.248669 | controller |     - '          exit 1'
2026-03-14 09:44:27.248674 | controller |     - '        fi'
2026-03-14 09:44:27.248680 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.248685 | controller |     - '        '
2026-03-14 09:44:27.248691 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.248697 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.248702 | controller |       | base64 -d)'
2026-03-14 09:44:27.248706 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.248710 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.248715 | controller |     - '          exit 1'
2026-03-14 09:44:27.248719 | controller |     - '        fi'
2026-03-14 09:44:27.248723 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.248727 | controller |     - '        '
2026-03-14 09:44:27.248731 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.248736 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.248740 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.248746 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.248750 | controller |       --timeout=300s'
2026-03-14 09:44:27.248756 | controller |     - '        '
2026-03-14 09:44:27.248767 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.248772 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.248776 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.248780 | controller |     - '        '
2026-03-14 09:44:27.248784 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.248789 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.248793 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.248797 | controller |     - '        '
2026-03-14 09:44:27.248801 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.248805 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.248810 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.248814 | controller |       available"'
2026-03-14 09:44:27.248818 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.248822 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.248826 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.248830 | controller |     - '          exit 1'
2026-03-14 09:44:27.248835 | controller |     - '        fi'
2026-03-14 09:44:27.248839 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.248843 | controller |     - '        '
2026-03-14 09:44:27.248847 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.248852 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.248856 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.248860 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.248864 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.248868 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.248873 | controller |       -- \'
2026-03-14 09:44:27.248880 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.248884 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.248888 | controller |     - '    logger.go:42: 09:26:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.248893 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.248898 | controller |     - '    logger.go:42: 09:26:49 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.248903 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.248907 | controller |       found'
2026-03-14 09:44:27.248911 | controller |     - '    logger.go:42: 09:26:50 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.248915 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.248919 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.248924 | controller |     - '        '
2026-03-14 09:44:27.248928 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.248934 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.248938 | controller |     - '        '
2026-03-14 09:44:27.248942 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.248946 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.248951 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.248955 | controller |     - '          exit 1'
2026-03-14 09:44:27.248960 | controller |     - '        fi'
2026-03-14 09:44:27.248965 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.248970 | controller |     - '        '
2026-03-14 09:44:27.248974 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.248979 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.248985 | controller |       | base64 -d)'
2026-03-14 09:44:27.248989 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.248993 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.248997 | controller |     - '          exit 1'
2026-03-14 09:44:27.249002 | controller |     - '        fi'
2026-03-14 09:44:27.249006 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.249010 | controller |     - '        '
2026-03-14 09:44:27.249014 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.249018 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.249023 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.249027 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.249031 | controller |       --timeout=300s'
2026-03-14 09:44:27.249035 | controller |     - '        '
2026-03-14 09:44:27.249044 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.249050 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.249054 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.249058 | controller |     - '        '
2026-03-14 09:44:27.249063 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.249067 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.249071 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.249075 | controller |     - '        '
2026-03-14 09:44:27.249079 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.249084 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.249088 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.249092 | controller |       available"'
2026-03-14 09:44:27.249096 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.249100 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.249105 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.249109 | controller |     - '          exit 1'
2026-03-14 09:44:27.249113 | controller |     - '        fi'
2026-03-14 09:44:27.249117 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.249121 | controller |     - '        '
2026-03-14 09:44:27.249125 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.249130 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.249135 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.249140 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.249146 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.249150 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.249155 | controller |       -- \'
2026-03-14 09:44:27.249159 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.249163 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.249167 | controller |     - '    logger.go:42: 09:26:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.249171 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.249176 | controller |     - '    logger.go:42: 09:26:50 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.249180 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.249184 | controller |       found'
2026-03-14 09:44:27.249188 | controller |     - '    logger.go:42: 09:26:51 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.249192 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.249197 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.249201 | controller |     - '        '
2026-03-14 09:44:27.249205 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.249209 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.249213 | controller |     - '        '
2026-03-14 09:44:27.249217 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.249222 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.249226 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.249230 | controller |     - '          exit 1'
2026-03-14 09:44:27.249234 | controller |     - '        fi'
2026-03-14 09:44:27.249238 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.249244 | controller |     - '        '
2026-03-14 09:44:27.249251 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.249255 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.249259 | controller |       | base64 -d)'
2026-03-14 09:44:27.249264 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.249268 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.249272 | controller |     - '          exit 1'
2026-03-14 09:44:27.249276 | controller |     - '        fi'
2026-03-14 09:44:27.249280 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.249284 | controller |     - '        '
2026-03-14 09:44:27.249304 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.249312 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.249316 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.249321 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.249325 | controller |       --timeout=300s'
2026-03-14 09:44:27.249329 | controller |     - '        '
2026-03-14 09:44:27.249333 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.249343 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.249348 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.249354 | controller |     - '        '
2026-03-14 09:44:27.249358 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.249362 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.249366 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.249371 | controller |     - '        '
2026-03-14 09:44:27.249375 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.249379 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.249383 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.249387 | controller |       available"'
2026-03-14 09:44:27.249392 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.249398 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.249403 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.249407 | controller |     - '          exit 1'
2026-03-14 09:44:27.249411 | controller |     - '        fi'
2026-03-14 09:44:27.249415 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.249419 | controller |     - '        '
2026-03-14 09:44:27.249424 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.249428 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.249432 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.249436 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.249440 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.249444 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.249449 | controller |       -- \'
2026-03-14 09:44:27.249453 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.249457 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.249461 | controller |     - '    logger.go:42: 09:26:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.249465 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.249470 | controller |     - '    logger.go:42: 09:26:51 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.249474 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.249478 | controller |       found'
2026-03-14 09:44:27.249482 | controller |     - '    logger.go:42: 09:26:52 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.249486 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.249490 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.249495 | controller |     - '        '
2026-03-14 09:44:27.249499 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.249503 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.249507 | controller |     - '        '
2026-03-14 09:44:27.249511 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.249515 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.249520 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.249524 | controller |     - '          exit 1'
2026-03-14 09:44:27.249528 | controller |     - '        fi'
2026-03-14 09:44:27.249534 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.249538 | controller |     - '        '
2026-03-14 09:44:27.249542 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.249546 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.249551 | controller |       | base64 -d)'
2026-03-14 09:44:27.249555 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.249559 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.249563 | controller |     - '          exit 1'
2026-03-14 09:44:27.249567 | controller |     - '        fi'
2026-03-14 09:44:27.249572 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.249576 | controller |     - '        '
2026-03-14 09:44:27.249580 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.249584 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.249588 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.249593 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.249597 | controller |       --timeout=300s'
2026-03-14 09:44:27.249601 | controller |     - '        '
2026-03-14 09:44:27.249605 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.249615 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.249622 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.249628 | controller |     - '        '
2026-03-14 09:44:27.249633 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.249639 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.249645 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.249651 | controller |     - '        '
2026-03-14 09:44:27.249657 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.249663 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.249668 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.249674 | controller |       available"'
2026-03-14 09:44:27.249679 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.249685 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.249690 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.249694 | controller |     - '          exit 1'
2026-03-14 09:44:27.249699 | controller |     - '        fi'
2026-03-14 09:44:27.249703 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.249707 | controller |     - '        '
2026-03-14 09:44:27.249711 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.249716 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.249720 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.249724 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.249728 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.249732 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.249740 | controller |       -- \'
2026-03-14 09:44:27.249744 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.249749 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.249755 | controller |     - '    logger.go:42: 09:26:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.249759 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.249763 | controller |     - '    logger.go:42: 09:26:52 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.249767 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.249772 | controller |       found'
2026-03-14 09:44:27.249776 | controller |     - '    logger.go:42: 09:26:53 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.249780 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.249784 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.249804 | controller |     - '        '
2026-03-14 09:44:27.249810 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.249814 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.249819 | controller |     - '        '
2026-03-14 09:44:27.249823 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.249827 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.249831 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.249835 | controller |     - '          exit 1'
2026-03-14 09:44:27.249840 | controller |     - '        fi'
2026-03-14 09:44:27.249844 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.249848 | controller |     - '        '
2026-03-14 09:44:27.249852 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.249856 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.249861 | controller |       | base64 -d)'
2026-03-14 09:44:27.249865 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.249869 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.249873 | controller |     - '          exit 1'
2026-03-14 09:44:27.249877 | controller |     - '        fi'
2026-03-14 09:44:27.249882 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.249886 | controller |     - '        '
2026-03-14 09:44:27.249890 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.249894 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.249903 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.249907 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.249912 | controller |       --timeout=300s'
2026-03-14 09:44:27.249916 | controller |     - '        '
2026-03-14 09:44:27.249920 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.249924 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.249936 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.249941 | controller |     - '        '
2026-03-14 09:44:27.249947 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.249951 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.249956 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.249960 | controller |     - '        '
2026-03-14 09:44:27.249966 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.249970 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.249974 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.249979 | controller |       available"'
2026-03-14 09:44:27.249983 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.249987 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.249991 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.249996 | controller |     - '          exit 1'
2026-03-14 09:44:27.250000 | controller |     - '        fi'
2026-03-14 09:44:27.250004 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.250008 | controller |     - '        '
2026-03-14 09:44:27.250012 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.250016 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.250021 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.250025 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.250029 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.250033 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.250037 | controller |       -- \'
2026-03-14 09:44:27.250041 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.250046 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.250050 | controller |     - '    logger.go:42: 09:26:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.250054 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.250058 | controller |     - '    logger.go:42: 09:26:53 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.250062 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.250066 | controller |       found'
2026-03-14 09:44:27.250071 | controller |     - '    logger.go:42: 09:26:54 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.250075 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.250079 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.250083 | controller |     - '        '
2026-03-14 09:44:27.250087 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.250091 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.250096 | controller |     - '        '
2026-03-14 09:44:27.250100 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.250104 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.250108 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.250112 | controller |     - '          exit 1'
2026-03-14 09:44:27.250116 | controller |     - '        fi'
2026-03-14 09:44:27.250121 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.250125 | controller |     - '        '
2026-03-14 09:44:27.250129 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.250133 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.250137 | controller |       | base64 -d)'
2026-03-14 09:44:27.250142 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.250146 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.250152 | controller |     - '          exit 1'
2026-03-14 09:44:27.250156 | controller |     - '        fi'
2026-03-14 09:44:27.250160 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.250164 | controller |     - '        '
2026-03-14 09:44:27.250168 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.250173 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.250177 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.250181 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.250185 | controller |       --timeout=300s'
2026-03-14 09:44:27.250189 | controller |     - '        '
2026-03-14 09:44:27.250193 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.250198 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.250208 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.250212 | controller |     - '        '
2026-03-14 09:44:27.250216 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.250222 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.250227 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.250231 | controller |     - '        '
2026-03-14 09:44:27.250235 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.250239 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.250243 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.250248 | controller |       available"'
2026-03-14 09:44:27.250252 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.250256 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.250260 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.250264 | controller |     - '          exit 1'
2026-03-14 09:44:27.250268 | controller |     - '        fi'
2026-03-14 09:44:27.250273 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.250277 | controller |     - '        '
2026-03-14 09:44:27.250281 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.250285 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.250303 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.250308 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.250312 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.250317 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.250321 | controller |       -- \'
2026-03-14 09:44:27.250326 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.250330 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.250334 | controller |     - '    logger.go:42: 09:26:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.250341 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.250345 | controller |     - '    logger.go:42: 09:26:54 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.250351 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.250372 | controller |       found'
2026-03-14 09:44:27.250376 | controller |     - '    logger.go:42: 09:26:55 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.250381 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.250385 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.250389 | controller |     - '        '
2026-03-14 09:44:27.250393 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.250398 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.250402 | controller |     - '        '
2026-03-14 09:44:27.250406 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.250411 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.250415 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.250419 | controller |     - '          exit 1'
2026-03-14 09:44:27.250423 | controller |     - '        fi'
2026-03-14 09:44:27.250428 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.250432 | controller |     - '        '
2026-03-14 09:44:27.250436 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.250440 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.250445 | controller |       | base64 -d)'
2026-03-14 09:44:27.250449 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.250453 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.250457 | controller |     - '          exit 1'
2026-03-14 09:44:27.250462 | controller |     - '        fi'
2026-03-14 09:44:27.250466 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.250470 | controller |     - '        '
2026-03-14 09:44:27.250474 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.250479 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.250483 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.250489 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.250493 | controller |       --timeout=300s'
2026-03-14 09:44:27.250497 | controller |     - '        '
2026-03-14 09:44:27.250502 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.250506 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.250517 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.250522 | controller |     - '        '
2026-03-14 09:44:27.250526 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.250530 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.250535 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.250540 | controller |     - '        '
2026-03-14 09:44:27.250544 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.250548 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.250552 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.250557 | controller |       available"'
2026-03-14 09:44:27.250561 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.250565 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.250572 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.250576 | controller |     - '          exit 1'
2026-03-14 09:44:27.250580 | controller |     - '        fi'
2026-03-14 09:44:27.250584 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.250589 | controller |     - '        '
2026-03-14 09:44:27.250593 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.250597 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.250601 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.250605 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.250610 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.250615 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.250620 | controller |       -- \'
2026-03-14 09:44:27.250626 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.250632 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.250638 | controller |     - '    logger.go:42: 09:26:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.250644 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.250649 | controller |     - '    logger.go:42: 09:26:56 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.250654 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.250658 | controller |       found'
2026-03-14 09:44:27.250662 | controller |     - '    logger.go:42: 09:26:57 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.250666 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.250670 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.250675 | controller |     - '        '
2026-03-14 09:44:27.250679 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.250683 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.250687 | controller |     - '        '
2026-03-14 09:44:27.250691 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.250696 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.250700 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.250704 | controller |     - '          exit 1'
2026-03-14 09:44:27.250708 | controller |     - '        fi'
2026-03-14 09:44:27.250712 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.250717 | controller |     - '        '
2026-03-14 09:44:27.250724 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.250728 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.250732 | controller |       | base64 -d)'
2026-03-14 09:44:27.250737 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.250741 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.250745 | controller |     - '          exit 1'
2026-03-14 09:44:27.250749 | controller |     - '        fi'
2026-03-14 09:44:27.250753 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.250758 | controller |     - '        '
2026-03-14 09:44:27.250762 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.250766 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.250770 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.250774 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.250780 | controller |       --timeout=300s'
2026-03-14 09:44:27.250785 | controller |     - '        '
2026-03-14 09:44:27.250789 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.250793 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.250797 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.250801 | controller |     - '        '
2026-03-14 09:44:27.250813 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.250817 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.250821 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.250826 | controller |     - '        '
2026-03-14 09:44:27.250830 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.250834 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.250838 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.250842 | controller |       available"'
2026-03-14 09:44:27.250847 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.250851 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.250855 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.250859 | controller |     - '          exit 1'
2026-03-14 09:44:27.250863 | controller |     - '        fi'
2026-03-14 09:44:27.250867 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.250872 | controller |     - '        '
2026-03-14 09:44:27.250876 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.250880 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.250884 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.250888 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.250892 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.250896 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.250901 | controller |       -- \'
2026-03-14 09:44:27.250905 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.250909 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.250913 | controller |     - '    logger.go:42: 09:26:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.250918 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.250923 | controller |     - '    logger.go:42: 09:26:58 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.250928 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.250933 | controller |       found'
2026-03-14 09:44:27.250937 | controller |     - '    logger.go:42: 09:26:59 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.250942 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.250948 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.250952 | controller |     - '        '
2026-03-14 09:44:27.250956 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.250960 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.250966 | controller |     - '        '
2026-03-14 09:44:27.250970 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.250974 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.250979 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.250983 | controller |     - '          exit 1'
2026-03-14 09:44:27.250987 | controller |     - '        fi'
2026-03-14 09:44:27.250991 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.250995 | controller |     - '        '
2026-03-14 09:44:27.250999 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.251004 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.251008 | controller |       | base64 -d)'
2026-03-14 09:44:27.251012 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.251016 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.251020 | controller |     - '          exit 1'
2026-03-14 09:44:27.251026 | controller |     - '        fi'
2026-03-14 09:44:27.251031 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.251035 | controller |     - '        '
2026-03-14 09:44:27.251039 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.251043 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.251047 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.251051 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.251056 | controller |       --timeout=300s'
2026-03-14 09:44:27.251060 | controller |     - '        '
2026-03-14 09:44:27.251064 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.251068 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.251072 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.251076 | controller |     - '        '
2026-03-14 09:44:27.251090 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.251095 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.251099 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.251104 | controller |     - '        '
2026-03-14 09:44:27.251108 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.251112 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.251116 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.251120 | controller |       available"'
2026-03-14 09:44:27.251125 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.251129 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.251133 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.251137 | controller |     - '          exit 1'
2026-03-14 09:44:27.251141 | controller |     - '        fi'
2026-03-14 09:44:27.251145 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.251149 | controller |     - '        '
2026-03-14 09:44:27.251154 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.251158 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.251162 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.251168 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.251172 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.251176 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.251182 | controller |       -- \'
2026-03-14 09:44:27.251186 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.251190 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.251194 | controller |     - '    logger.go:42: 09:26:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.251199 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.251203 | controller |     - '    logger.go:42: 09:26:59 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.251207 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.251211 | controller |       found'
2026-03-14 09:44:27.251215 | controller |     - '    logger.go:42: 09:27:00 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.251219 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.251224 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.251228 | controller |     - '        '
2026-03-14 09:44:27.251232 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.251236 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.251240 | controller |     - '        '
2026-03-14 09:44:27.251244 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.251249 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.251253 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.251257 | controller |     - '          exit 1'
2026-03-14 09:44:27.251261 | controller |     - '        fi'
2026-03-14 09:44:27.251265 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.251269 | controller |     - '        '
2026-03-14 09:44:27.251274 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.251278 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.251282 | controller |       | base64 -d)'
2026-03-14 09:44:27.251286 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.251306 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.251311 | controller |     - '          exit 1'
2026-03-14 09:44:27.251316 | controller |     - '        fi'
2026-03-14 09:44:27.251320 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.251324 | controller |     - '        '
2026-03-14 09:44:27.251328 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.251332 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.251337 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.251341 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.251345 | controller |       --timeout=300s'
2026-03-14 09:44:27.251349 | controller |     - '        '
2026-03-14 09:44:27.251353 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.251358 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.251362 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.251368 | controller |     - '        '
2026-03-14 09:44:27.251372 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.251382 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.251387 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.251391 | controller |     - '        '
2026-03-14 09:44:27.251395 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.251399 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.251404 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.251408 | controller |       available"'
2026-03-14 09:44:27.251412 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.251416 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.251420 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.251425 | controller |     - '          exit 1'
2026-03-14 09:44:27.251429 | controller |     - '        fi'
2026-03-14 09:44:27.251433 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.251437 | controller |     - '        '
2026-03-14 09:44:27.251441 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.251445 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.251450 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.251454 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.251458 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.251462 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.251466 | controller |       -- \'
2026-03-14 09:44:27.251471 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.251475 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.251479 | controller |     - '    logger.go:42: 09:27:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.251483 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.251487 | controller |     - '    logger.go:42: 09:27:00 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.251491 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.251496 | controller |       found'
2026-03-14 09:44:27.251500 | controller |     - '    logger.go:42: 09:27:01 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.251504 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.251508 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.251512 | controller |     - '        '
2026-03-14 09:44:27.251516 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.251521 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.251525 | controller |     - '        '
2026-03-14 09:44:27.251529 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.251533 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.251537 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.251542 | controller |     - '          exit 1'
2026-03-14 09:44:27.251546 | controller |     - '        fi'
2026-03-14 09:44:27.251550 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.251556 | controller |     - '        '
2026-03-14 09:44:27.251560 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.251564 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.251568 | controller |       | base64 -d)'
2026-03-14 09:44:27.251573 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.251577 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.251581 | controller |     - '          exit 1'
2026-03-14 09:44:27.251585 | controller |     - '        fi'
2026-03-14 09:44:27.251589 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.251594 | controller |     - '        '
2026-03-14 09:44:27.251598 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.251602 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.251606 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.251610 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.251616 | controller |       --timeout=300s'
2026-03-14 09:44:27.251621 | controller |     - '        '
2026-03-14 09:44:27.251627 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.251633 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.251639 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.251645 | controller |     - '        '
2026-03-14 09:44:27.251650 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.251664 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.251669 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.251674 | controller |     - '        '
2026-03-14 09:44:27.251678 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.251682 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.251687 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.251691 | controller |       available"'
2026-03-14 09:44:27.251695 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.251699 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.251703 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.251707 | controller |     - '          exit 1'
2026-03-14 09:44:27.251712 | controller |     - '        fi'
2026-03-14 09:44:27.251716 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.251720 | controller |     - '        '
2026-03-14 09:44:27.251724 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.251728 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.251733 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.251737 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.251741 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.251745 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.251749 | controller |       -- \'
2026-03-14 09:44:27.251753 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.251781 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.251785 | controller |     - '    logger.go:42: 09:27:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.251789 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.251793 | controller |     - '    logger.go:42: 09:27:01 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.251797 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.251802 | controller |       found'
2026-03-14 09:44:27.251806 | controller |     - '    logger.go:42: 09:27:02 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.251810 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.251814 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.251818 | controller |     - '        '
2026-03-14 09:44:27.251823 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.251827 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.251831 | controller |     - '        '
2026-03-14 09:44:27.251835 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.251839 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.251843 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.251848 | controller |     - '          exit 1'
2026-03-14 09:44:27.251852 | controller |     - '        fi'
2026-03-14 09:44:27.251856 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.251860 | controller |     - '        '
2026-03-14 09:44:27.251864 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.251868 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.251872 | controller |       | base64 -d)'
2026-03-14 09:44:27.251877 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.251881 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.251885 | controller |     - '          exit 1'
2026-03-14 09:44:27.251889 | controller |     - '        fi'
2026-03-14 09:44:27.251893 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.251897 | controller |     - '        '
2026-03-14 09:44:27.251902 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.251906 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.251910 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.251914 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.251918 | controller |       --timeout=300s'
2026-03-14 09:44:27.251922 | controller |     - '        '
2026-03-14 09:44:27.251926 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.251931 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.251935 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.251939 | controller |     - '        '
2026-03-14 09:44:27.251943 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.251948 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.251957 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.251961 | controller |     - '        '
2026-03-14 09:44:27.251966 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.251972 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.251976 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.251980 | controller |       available"'
2026-03-14 09:44:27.251984 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.251990 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.251995 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.251999 | controller |     - '          exit 1'
2026-03-14 09:44:27.252003 | controller |     - '        fi'
2026-03-14 09:44:27.252007 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.252011 | controller |     - '        '
2026-03-14 09:44:27.252015 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.252020 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.252024 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.252028 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.252032 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.252036 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.252040 | controller |       -- \'
2026-03-14 09:44:27.252044 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.252048 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.252053 | controller |     - '    logger.go:42: 09:27:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.252057 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.252061 | controller |     - '    logger.go:42: 09:27:02 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.252065 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.252069 | controller |       found'
2026-03-14 09:44:27.252074 | controller |     - '    logger.go:42: 09:27:03 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.252078 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.252084 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.252088 | controller |     - '        '
2026-03-14 09:44:27.252092 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.252097 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.252101 | controller |     - '        '
2026-03-14 09:44:27.252107 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.252111 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.252115 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.252120 | controller |     - '          exit 1'
2026-03-14 09:44:27.252124 | controller |     - '        fi'
2026-03-14 09:44:27.252128 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.252132 | controller |     - '        '
2026-03-14 09:44:27.252137 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.252141 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.252145 | controller |       | base64 -d)'
2026-03-14 09:44:27.252149 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.252153 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.252157 | controller |     - '          exit 1'
2026-03-14 09:44:27.252163 | controller |     - '        fi'
2026-03-14 09:44:27.252167 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.252172 | controller |     - '        '
2026-03-14 09:44:27.252176 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.252180 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.252184 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.252188 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.252192 | controller |       --timeout=300s'
2026-03-14 09:44:27.252197 | controller |     - '        '
2026-03-14 09:44:27.252201 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.252205 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.252209 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.252214 | controller |     - '        '
2026-03-14 09:44:27.252218 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.252222 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.252234 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.252239 | controller |     - '        '
2026-03-14 09:44:27.252244 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.252248 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.252252 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.252256 | controller |       available"'
2026-03-14 09:44:27.252261 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.252267 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.252271 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.252275 | controller |     - '          exit 1'
2026-03-14 09:44:27.252279 | controller |     - '        fi'
2026-03-14 09:44:27.252284 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.252301 | controller |     - '        '
2026-03-14 09:44:27.252308 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.252314 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.252318 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.252322 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.252327 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.252331 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.252335 | controller |       -- \'
2026-03-14 09:44:27.252340 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.252344 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.252348 | controller |     - '    logger.go:42: 09:27:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.252352 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.252357 | controller |     - '    logger.go:42: 09:27:03 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.252361 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.252368 | controller |       found'
2026-03-14 09:44:27.252373 | controller |     - '    logger.go:42: 09:27:04 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.252377 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.252381 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.252386 | controller |     - '        '
2026-03-14 09:44:27.252390 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.252394 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.252398 | controller |     - '        '
2026-03-14 09:44:27.252403 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.252407 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.252411 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.252415 | controller |     - '          exit 1'
2026-03-14 09:44:27.252420 | controller |     - '        fi'
2026-03-14 09:44:27.252424 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.252428 | controller |     - '        '
2026-03-14 09:44:27.252432 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.252436 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.252441 | controller |       | base64 -d)'
2026-03-14 09:44:27.252445 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.252452 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.252456 | controller |     - '          exit 1'
2026-03-14 09:44:27.252460 | controller |     - '        fi'
2026-03-14 09:44:27.252464 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.252470 | controller |     - '        '
2026-03-14 09:44:27.252475 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.252482 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.252487 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.252493 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.252499 | controller |       --timeout=300s'
2026-03-14 09:44:27.252505 | controller |     - '        '
2026-03-14 09:44:27.252511 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.252517 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.252523 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.252528 | controller |     - '        '
2026-03-14 09:44:27.252534 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.252539 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.252545 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.252560 | controller |     - '        '
2026-03-14 09:44:27.252568 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.252574 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.252579 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.252584 | controller |       available"'
2026-03-14 09:44:27.252588 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.252592 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.252596 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.252603 | controller |     - '          exit 1'
2026-03-14 09:44:27.252608 | controller |     - '        fi'
2026-03-14 09:44:27.252612 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.252618 | controller |     - '        '
2026-03-14 09:44:27.252624 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.252630 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.252636 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.252642 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.252648 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.252653 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.252657 | controller |       -- \'
2026-03-14 09:44:27.252661 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.252665 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.252670 | controller |     - '    logger.go:42: 09:27:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.252674 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.252678 | controller |     - '    logger.go:42: 09:27:04 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.252682 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.252686 | controller |       found'
2026-03-14 09:44:27.252690 | controller |     - '    logger.go:42: 09:27:05 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.252694 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.252699 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.252703 | controller |     - '        '
2026-03-14 09:44:27.252707 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.252711 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.252715 | controller |     - '        '
2026-03-14 09:44:27.252719 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.252724 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.252728 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.252732 | controller |     - '          exit 1'
2026-03-14 09:44:27.252736 | controller |     - '        fi'
2026-03-14 09:44:27.252743 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.252747 | controller |     - '        '
2026-03-14 09:44:27.252752 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.252756 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.252760 | controller |       | base64 -d)'
2026-03-14 09:44:27.252764 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.252768 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.252772 | controller |     - '          exit 1'
2026-03-14 09:44:27.252777 | controller |     - '        fi'
2026-03-14 09:44:27.252781 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.252785 | controller |     - '        '
2026-03-14 09:44:27.252789 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.252794 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.252798 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.252802 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.252808 | controller |       --timeout=300s'
2026-03-14 09:44:27.252814 | controller |     - '        '
2026-03-14 09:44:27.252818 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.252822 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.252826 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.252831 | controller |     - '        '
2026-03-14 09:44:27.252835 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.252839 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.252843 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.252847 | controller |     - '        '
2026-03-14 09:44:27.252859 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.252864 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.252868 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.252872 | controller |       available"'
2026-03-14 09:44:27.252876 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.252880 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.252885 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.252889 | controller |     - '          exit 1'
2026-03-14 09:44:27.252893 | controller |     - '        fi'
2026-03-14 09:44:27.252897 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.252901 | controller |     - '        '
2026-03-14 09:44:27.252906 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.252910 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.252914 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.252918 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.252922 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.252926 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.252931 | controller |       -- \'
2026-03-14 09:44:27.252935 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.252939 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.252943 | controller |     - '    logger.go:42: 09:27:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.252947 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.252953 | controller |     - '    logger.go:42: 09:27:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.252958 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.252962 | controller |       found'
2026-03-14 09:44:27.252966 | controller |     - '    logger.go:42: 09:27:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.252970 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.252974 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.252979 | controller |     - '        '
2026-03-14 09:44:27.252983 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.252987 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.252993 | controller |     - '        '
2026-03-14 09:44:27.252997 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.253001 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.253005 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.253010 | controller |     - '          exit 1'
2026-03-14 09:44:27.253014 | controller |     - '        fi'
2026-03-14 09:44:27.253018 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.253022 | controller |     - '        '
2026-03-14 09:44:27.253026 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.253031 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.253035 | controller |       | base64 -d)'
2026-03-14 09:44:27.253039 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.253043 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.253047 | controller |     - '          exit 1'
2026-03-14 09:44:27.253051 | controller |     - '        fi'
2026-03-14 09:44:27.253056 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.253060 | controller |     - '        '
2026-03-14 09:44:27.253064 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.253068 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.253072 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.253077 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.253081 | controller |       --timeout=300s'
2026-03-14 09:44:27.253085 | controller |     - '        '
2026-03-14 09:44:27.253089 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.253093 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.253097 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.253102 | controller |     - '        '
2026-03-14 09:44:27.253106 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.253110 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.253114 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.253118 | controller |     - '        '
2026-03-14 09:44:27.253123 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.253132 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.253137 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.253141 | controller |       available"'
2026-03-14 09:44:27.253147 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.253152 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.253156 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.253160 | controller |     - '          exit 1'
2026-03-14 09:44:27.253164 | controller |     - '        fi'
2026-03-14 09:44:27.253168 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.253173 | controller |     - '        '
2026-03-14 09:44:27.253177 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.253181 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.253185 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.253191 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.253195 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.253199 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.253204 | controller |       -- \'
2026-03-14 09:44:27.253208 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.253212 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.253218 | controller |     - '    logger.go:42: 09:27:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.253222 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.253226 | controller |     - '    logger.go:42: 09:27:07 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.253230 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.253235 | controller |       found'
2026-03-14 09:44:27.253239 | controller |     - '    logger.go:42: 09:27:08 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.253243 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.253247 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.253251 | controller |     - '        '
2026-03-14 09:44:27.253256 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.253260 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.253264 | controller |     - '        '
2026-03-14 09:44:27.253268 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.253272 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.253277 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.253281 | controller |     - '          exit 1'
2026-03-14 09:44:27.253285 | controller |     - '        fi'
2026-03-14 09:44:27.253315 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.253343 | controller |     - '        '
2026-03-14 09:44:27.253348 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.253352 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.253356 | controller |       | base64 -d)'
2026-03-14 09:44:27.253360 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.253364 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.253371 | controller |     - '          exit 1'
2026-03-14 09:44:27.253375 | controller |     - '        fi'
2026-03-14 09:44:27.253380 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.253384 | controller |     - '        '
2026-03-14 09:44:27.253388 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.253392 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.253396 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.253400 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.253405 | controller |       --timeout=300s'
2026-03-14 09:44:27.253409 | controller |     - '        '
2026-03-14 09:44:27.253413 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.253417 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.253421 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.253426 | controller |     - '        '
2026-03-14 09:44:27.253432 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.253436 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.253440 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.253444 | controller |     - '        '
2026-03-14 09:44:27.253448 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.253460 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.253464 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.253468 | controller |       available"'
2026-03-14 09:44:27.253473 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.253477 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.253481 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.253485 | controller |     - '          exit 1'
2026-03-14 09:44:27.253489 | controller |     - '        fi'
2026-03-14 09:44:27.253493 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.253498 | controller |     - '        '
2026-03-14 09:44:27.253502 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.253506 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.253510 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.253516 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.253520 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.253524 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.253529 | controller |       -- \'
2026-03-14 09:44:27.253533 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.253537 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.253541 | controller |     - '    logger.go:42: 09:27:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.253545 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.253550 | controller |     - '    logger.go:42: 09:27:08 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.253554 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.253558 | controller |       found'
2026-03-14 09:44:27.253562 | controller |     - '    logger.go:42: 09:27:09 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.253566 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.253571 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.253575 | controller |     - '        '
2026-03-14 09:44:27.253579 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.253583 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.253587 | controller |     - '        '
2026-03-14 09:44:27.253591 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.253595 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.253600 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.253604 | controller |     - '          exit 1'
2026-03-14 09:44:27.253608 | controller |     - '        fi'
2026-03-14 09:44:27.253613 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.253618 | controller |     - '        '
2026-03-14 09:44:27.253624 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.253633 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.253640 | controller |       | base64 -d)'
2026-03-14 09:44:27.253646 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.253650 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.253654 | controller |     - '          exit 1'
2026-03-14 09:44:27.253658 | controller |     - '        fi'
2026-03-14 09:44:27.253662 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.253666 | controller |     - '        '
2026-03-14 09:44:27.253671 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.253675 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.253679 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.253683 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.253687 | controller |       --timeout=300s'
2026-03-14 09:44:27.253691 | controller |     - '        '
2026-03-14 09:44:27.253695 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.253699 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.253704 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.253708 | controller |     - '        '
2026-03-14 09:44:27.253712 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.253716 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.253720 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.253724 | controller |     - '        '
2026-03-14 09:44:27.253728 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.253733 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.253743 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.253747 | controller |       available"'
2026-03-14 09:44:27.253751 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.253756 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.253760 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.253764 | controller |     - '          exit 1'
2026-03-14 09:44:27.253768 | controller |     - '        fi'
2026-03-14 09:44:27.253772 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.253777 | controller |     - '        '
2026-03-14 09:44:27.253781 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.253785 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.253789 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.253793 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.253810 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.253816 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.253822 | controller |       -- \'
2026-03-14 09:44:27.253827 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.253835 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.253842 | controller |     - '    logger.go:42: 09:27:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.253846 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.253851 | controller |     - '    logger.go:42: 09:27:09 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.253855 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.253859 | controller |       found'
2026-03-14 09:44:27.253863 | controller |     - '    logger.go:42: 09:27:10 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.253867 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.253872 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.253876 | controller |     - '        '
2026-03-14 09:44:27.253880 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.253884 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.253888 | controller |     - '        '
2026-03-14 09:44:27.253892 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.253897 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.253901 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.253905 | controller |     - '          exit 1'
2026-03-14 09:44:27.253910 | controller |     - '        fi'
2026-03-14 09:44:27.253914 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.253918 | controller |     - '        '
2026-03-14 09:44:27.253924 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.253929 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.253933 | controller |       | base64 -d)'
2026-03-14 09:44:27.253938 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.253943 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.253948 | controller |     - '          exit 1'
2026-03-14 09:44:27.253952 | controller |     - '        fi'
2026-03-14 09:44:27.253956 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.253961 | controller |     - '        '
2026-03-14 09:44:27.253965 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.253969 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.253973 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.253977 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.253981 | controller |       --timeout=300s'
2026-03-14 09:44:27.253986 | controller |     - '        '
2026-03-14 09:44:27.253990 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.253994 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.253998 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.254002 | controller |     - '        '
2026-03-14 09:44:27.254007 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.254012 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.254016 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.254020 | controller |     - '        '
2026-03-14 09:44:27.254024 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.254029 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.254035 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.254045 | controller |       available"'
2026-03-14 09:44:27.254049 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.254054 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.254058 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.254062 | controller |     - '          exit 1'
2026-03-14 09:44:27.254066 | controller |     - '        fi'
2026-03-14 09:44:27.254070 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.254074 | controller |     - '        '
2026-03-14 09:44:27.254079 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.254083 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.254087 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.254091 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.254095 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.254099 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.254104 | controller |       -- \'
2026-03-14 09:44:27.254108 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.254114 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.254118 | controller |     - '    logger.go:42: 09:27:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.254122 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.254126 | controller |     - '    logger.go:42: 09:27:10 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.254131 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.254135 | controller |       found'
2026-03-14 09:44:27.254139 | controller |     - '    logger.go:42: 09:27:11 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.254143 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.254147 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.254151 | controller |     - '        '
2026-03-14 09:44:27.254156 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.254160 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.254164 | controller |     - '        '
2026-03-14 09:44:27.254168 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.254172 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.254176 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.254181 | controller |     - '          exit 1'
2026-03-14 09:44:27.254185 | controller |     - '        fi'
2026-03-14 09:44:27.254189 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.254193 | controller |     - '        '
2026-03-14 09:44:27.254197 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.254201 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.254206 | controller |       | base64 -d)'
2026-03-14 09:44:27.254210 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.254214 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.254218 | controller |     - '          exit 1'
2026-03-14 09:44:27.254222 | controller |     - '        fi'
2026-03-14 09:44:27.254226 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.254232 | controller |     - '        '
2026-03-14 09:44:27.254238 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.254242 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.254247 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.254251 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.254255 | controller |       --timeout=300s'
2026-03-14 09:44:27.254259 | controller |     - '        '
2026-03-14 09:44:27.254263 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.254267 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.254272 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.254276 | controller |     - '        '
2026-03-14 09:44:27.254280 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.254284 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.254302 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.254309 | controller |     - '        '
2026-03-14 09:44:27.254313 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.254320 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.254324 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.254328 | controller |       available"'
2026-03-14 09:44:27.254338 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.254343 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.254347 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.254351 | controller |     - '          exit 1'
2026-03-14 09:44:27.254355 | controller |     - '        fi'
2026-03-14 09:44:27.254359 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.254364 | controller |     - '        '
2026-03-14 09:44:27.254368 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.254372 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.254376 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.254380 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.254385 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.254389 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.254393 | controller |       -- \'
2026-03-14 09:44:27.254397 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.254401 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.254405 | controller |     - '    logger.go:42: 09:27:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.254410 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.254414 | controller |     - '    logger.go:42: 09:27:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.254418 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.254422 | controller |       found'
2026-03-14 09:44:27.254428 | controller |     - '    logger.go:42: 09:27:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.254432 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.254437 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.254441 | controller |     - '        '
2026-03-14 09:44:27.254445 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.254449 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.254453 | controller |     - '        '
2026-03-14 09:44:27.254457 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.254462 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.254466 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.254470 | controller |     - '          exit 1'
2026-03-14 09:44:27.254474 | controller |     - '        fi'
2026-03-14 09:44:27.254478 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.254482 | controller |     - '        '
2026-03-14 09:44:27.254487 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.254491 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.254495 | controller |       | base64 -d)'
2026-03-14 09:44:27.254499 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.254503 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.254507 | controller |     - '          exit 1'
2026-03-14 09:44:27.254512 | controller |     - '        fi'
2026-03-14 09:44:27.254516 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.254520 | controller |     - '        '
2026-03-14 09:44:27.254524 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.254528 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.254532 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.254537 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.254541 | controller |       --timeout=300s'
2026-03-14 09:44:27.254545 | controller |     - '        '
2026-03-14 09:44:27.254549 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.254553 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.254557 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.254562 | controller |     - '        '
2026-03-14 09:44:27.254566 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.254570 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.254574 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.254578 | controller |     - '        '
2026-03-14 09:44:27.254582 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.254587 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.254591 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.254595 | controller |       available"'
2026-03-14 09:44:27.254599 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.254609 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.254615 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.254621 | controller |     - '          exit 1'
2026-03-14 09:44:27.254629 | controller |     - '        fi'
2026-03-14 09:44:27.254635 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.254641 | controller |     - '        '
2026-03-14 09:44:27.254647 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.254651 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.254655 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.254660 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.254664 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.254668 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.254672 | controller |       -- \'
2026-03-14 09:44:27.254676 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.254680 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.254685 | controller |     - '    logger.go:42: 09:27:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.254689 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.254705 | controller |     - '    logger.go:42: 09:27:12 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.254709 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.254714 | controller |       found'
2026-03-14 09:44:27.254718 | controller |     - '    logger.go:42: 09:27:13 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.254722 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.254726 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.254730 | controller |     - '        '
2026-03-14 09:44:27.254735 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.254739 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.254743 | controller |     - '        '
2026-03-14 09:44:27.254747 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.254751 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.254756 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.254760 | controller |     - '          exit 1'
2026-03-14 09:44:27.254764 | controller |     - '        fi'
2026-03-14 09:44:27.254768 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.254772 | controller |     - '        '
2026-03-14 09:44:27.254776 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.254781 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.254785 | controller |       | base64 -d)'
2026-03-14 09:44:27.254789 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.254793 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.254797 | controller |     - '          exit 1'
2026-03-14 09:44:27.254801 | controller |     - '        fi'
2026-03-14 09:44:27.254806 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.254810 | controller |     - '        '
2026-03-14 09:44:27.254814 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.254818 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.254822 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.254826 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.254831 | controller |       --timeout=300s'
2026-03-14 09:44:27.254844 | controller |     - '        '
2026-03-14 09:44:27.254848 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.254852 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.254856 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.254860 | controller |     - '        '
2026-03-14 09:44:27.254865 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.254869 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.254873 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.254877 | controller |     - '        '
2026-03-14 09:44:27.254881 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.254886 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.254890 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.254894 | controller |       available"'
2026-03-14 09:44:27.254898 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.254902 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.254915 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.254921 | controller |     - '          exit 1'
2026-03-14 09:44:27.254925 | controller |     - '        fi'
2026-03-14 09:44:27.254930 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.254935 | controller |     - '        '
2026-03-14 09:44:27.254940 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.254944 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.254948 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.254952 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.254957 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.254961 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.254965 | controller |       -- \'
2026-03-14 09:44:27.254969 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.254973 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.254978 | controller |     - '    logger.go:42: 09:27:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.254982 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.254986 | controller |     - '    logger.go:42: 09:27:13 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.254990 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.254994 | controller |       found'
2026-03-14 09:44:27.254999 | controller |     - '    logger.go:42: 09:27:14 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.255003 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.255007 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.255011 | controller |     - '        '
2026-03-14 09:44:27.255015 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.255019 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.255024 | controller |     - '        '
2026-03-14 09:44:27.255030 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.255035 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.255039 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.255043 | controller |     - '          exit 1'
2026-03-14 09:44:27.255047 | controller |     - '        fi'
2026-03-14 09:44:27.255051 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.255056 | controller |     - '        '
2026-03-14 09:44:27.255060 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.255064 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.255068 | controller |       | base64 -d)'
2026-03-14 09:44:27.255072 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.255077 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.255081 | controller |     - '          exit 1'
2026-03-14 09:44:27.255085 | controller |     - '        fi'
2026-03-14 09:44:27.255089 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.255093 | controller |     - '        '
2026-03-14 09:44:27.255097 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.255102 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.255106 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.255110 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.255114 | controller |       --timeout=300s'
2026-03-14 09:44:27.255126 | controller |     - '        '
2026-03-14 09:44:27.255131 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.255135 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.255139 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.255143 | controller |     - '        '
2026-03-14 09:44:27.255149 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.255153 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.255157 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.255162 | controller |     - '        '
2026-03-14 09:44:27.255166 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.255170 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.255174 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.255178 | controller |       available"'
2026-03-14 09:44:27.255182 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.255187 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.255197 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.255201 | controller |     - '          exit 1'
2026-03-14 09:44:27.255205 | controller |     - '        fi'
2026-03-14 09:44:27.255210 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.255214 | controller |     - '        '
2026-03-14 09:44:27.255218 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.255222 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.255226 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.255230 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.255243 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.255248 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.255252 | controller |       -- \'
2026-03-14 09:44:27.255256 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.255260 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.255265 | controller |     - '    logger.go:42: 09:27:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.255269 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.255273 | controller |     - '    logger.go:42: 09:27:15 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.255280 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.255284 | controller |       found'
2026-03-14 09:44:27.255302 | controller |     - '    logger.go:42: 09:27:16 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.255308 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.255313 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.255317 | controller |     - '        '
2026-03-14 09:44:27.255321 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.255325 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.255329 | controller |     - '        '
2026-03-14 09:44:27.255334 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.255338 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.255342 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.255346 | controller |     - '          exit 1'
2026-03-14 09:44:27.255350 | controller |     - '        fi'
2026-03-14 09:44:27.255357 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.255361 | controller |     - '        '
2026-03-14 09:44:27.255365 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.255370 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.255374 | controller |       | base64 -d)'
2026-03-14 09:44:27.255378 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.255383 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.255387 | controller |     - '          exit 1'
2026-03-14 09:44:27.255391 | controller |     - '        fi'
2026-03-14 09:44:27.255395 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.255399 | controller |     - '        '
2026-03-14 09:44:27.255404 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.255408 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.255412 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.255417 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.255421 | controller |       --timeout=300s'
2026-03-14 09:44:27.255425 | controller |     - '        '
2026-03-14 09:44:27.255429 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.255434 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.255439 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.255444 | controller |     - '        '
2026-03-14 09:44:27.255448 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.255454 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.255458 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.255463 | controller |     - '        '
2026-03-14 09:44:27.255467 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.255471 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.255475 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.255479 | controller |       available"'
2026-03-14 09:44:27.255484 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.255488 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.255492 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.255496 | controller |     - '          exit 1'
2026-03-14 09:44:27.255508 | controller |     - '        fi'
2026-03-14 09:44:27.255513 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.255517 | controller |     - '        '
2026-03-14 09:44:27.255521 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.255525 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.255529 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.255534 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.255538 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.255542 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.255546 | controller |       -- \'
2026-03-14 09:44:27.255550 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.255555 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.255559 | controller |     - '    logger.go:42: 09:27:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.255563 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.255567 | controller |     - '    logger.go:42: 09:27:16 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.255571 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.255576 | controller |       found'
2026-03-14 09:44:27.255580 | controller |     - '    logger.go:42: 09:27:17 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.255584 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.255588 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.255593 | controller |     - '        '
2026-03-14 09:44:27.255597 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.255601 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.255605 | controller |     - '        '
2026-03-14 09:44:27.255609 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.255615 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.255623 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.255629 | controller |     - '          exit 1'
2026-03-14 09:44:27.255635 | controller |     - '        fi'
2026-03-14 09:44:27.255641 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.255647 | controller |     - '        '
2026-03-14 09:44:27.255653 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.255659 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.255664 | controller |       | base64 -d)'
2026-03-14 09:44:27.255668 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.255672 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.255676 | controller |     - '          exit 1'
2026-03-14 09:44:27.255680 | controller |     - '        fi'
2026-03-14 09:44:27.255685 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.255689 | controller |     - '        '
2026-03-14 09:44:27.255693 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.255697 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.255702 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.255706 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.255710 | controller |       --timeout=300s'
2026-03-14 09:44:27.255714 | controller |     - '        '
2026-03-14 09:44:27.255718 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.255723 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.255727 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.255731 | controller |     - '        '
2026-03-14 09:44:27.255735 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.255739 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.255744 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.255748 | controller |     - '        '
2026-03-14 09:44:27.255752 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.255756 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.255761 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.255765 | controller |       available"'
2026-03-14 09:44:27.255769 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.255773 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.255777 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.255781 | controller |     - '          exit 1'
2026-03-14 09:44:27.255786 | controller |     - '        fi'
2026-03-14 09:44:27.255797 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.255801 | controller |     - '        '
2026-03-14 09:44:27.255806 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.255810 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.255814 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.255818 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.255823 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.255827 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.255831 | controller |       -- \'
2026-03-14 09:44:27.255835 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.255840 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.255846 | controller |     - '    logger.go:42: 09:27:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.255850 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.255854 | controller |     - '    logger.go:42: 09:27:17 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.255859 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.255863 | controller |       found'
2026-03-14 09:44:27.255870 | controller |     - '    logger.go:42: 09:27:18 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.255874 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.255878 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.255883 | controller |     - '        '
2026-03-14 09:44:27.255887 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.255891 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.255895 | controller |     - '        '
2026-03-14 09:44:27.255899 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.255903 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.255908 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.255912 | controller |     - '          exit 1'
2026-03-14 09:44:27.255916 | controller |     - '        fi'
2026-03-14 09:44:27.255920 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.255924 | controller |     - '        '
2026-03-14 09:44:27.255930 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.255934 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.255938 | controller |       | base64 -d)'
2026-03-14 09:44:27.255943 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.255947 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.255951 | controller |     - '          exit 1'
2026-03-14 09:44:27.255955 | controller |     - '        fi'
2026-03-14 09:44:27.255959 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.255963 | controller |     - '        '
2026-03-14 09:44:27.255968 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.255972 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.255976 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.255980 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.255984 | controller |       --timeout=300s'
2026-03-14 09:44:27.255988 | controller |     - '        '
2026-03-14 09:44:27.255992 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.255996 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.256001 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.256005 | controller |     - '        '
2026-03-14 09:44:27.256009 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.256013 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.256017 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.256021 | controller |     - '        '
2026-03-14 09:44:27.256026 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.256030 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.256036 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.256040 | controller |       available"'
2026-03-14 09:44:27.256044 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.256048 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.256052 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.256057 | controller |     - '          exit 1'
2026-03-14 09:44:27.256061 | controller |     - '        fi'
2026-03-14 09:44:27.256065 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.256075 | controller |     - '        '
2026-03-14 09:44:27.256079 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.256084 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.256088 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.256092 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.256096 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.256100 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.256105 | controller |       -- \'
2026-03-14 09:44:27.256109 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.256113 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.256117 | controller |     - '    logger.go:42: 09:27:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.256121 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.256126 | controller |     - '    logger.go:42: 09:27:18 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.256130 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.256134 | controller |       found'
2026-03-14 09:44:27.256138 | controller |     - '    logger.go:42: 09:27:19 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.256142 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.256147 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.256151 | controller |     - '        '
2026-03-14 09:44:27.256155 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.256159 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.256163 | controller |     - '        '
2026-03-14 09:44:27.256167 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.256171 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.256176 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.256180 | controller |     - '          exit 1'
2026-03-14 09:44:27.256184 | controller |     - '        fi'
2026-03-14 09:44:27.256188 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.256192 | controller |     - '        '
2026-03-14 09:44:27.256197 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.256201 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.256205 | controller |       | base64 -d)'
2026-03-14 09:44:27.256209 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.256213 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.256218 | controller |     - '          exit 1'
2026-03-14 09:44:27.256222 | controller |     - '        fi'
2026-03-14 09:44:27.256226 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.256232 | controller |     - '        '
2026-03-14 09:44:27.256236 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.256240 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.256244 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.256249 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.256253 | controller |       --timeout=300s'
2026-03-14 09:44:27.256257 | controller |     - '        '
2026-03-14 09:44:27.256261 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.256265 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.256269 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.256274 | controller |     - '        '
2026-03-14 09:44:27.256278 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.256282 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.256289 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.256306 | controller |     - '        '
2026-03-14 09:44:27.256312 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.256316 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.256320 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.256324 | controller |       available"'
2026-03-14 09:44:27.256329 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.256333 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.256337 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.256341 | controller |     - '          exit 1'
2026-03-14 09:44:27.256345 | controller |     - '        fi'
2026-03-14 09:44:27.256352 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.256356 | controller |     - '        '
2026-03-14 09:44:27.256366 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.256371 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.256375 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.256379 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.256385 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.256389 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.256393 | controller |       -- \'
2026-03-14 09:44:27.256397 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.256402 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.256406 | controller |     - '    logger.go:42: 09:27:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.256410 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.256414 | controller |     - '    logger.go:42: 09:27:19 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.256418 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.256423 | controller |       found'
2026-03-14 09:44:27.256427 | controller |     - '    logger.go:42: 09:27:20 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.256433 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.256437 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.256441 | controller |     - '        '
2026-03-14 09:44:27.256456 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.256462 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.256466 | controller |     - '        '
2026-03-14 09:44:27.256470 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.256474 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.256479 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.256483 | controller |     - '          exit 1'
2026-03-14 09:44:27.256487 | controller |     - '        fi'
2026-03-14 09:44:27.256491 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.256495 | controller |     - '        '
2026-03-14 09:44:27.256500 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.256504 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.256508 | controller |       | base64 -d)'
2026-03-14 09:44:27.256512 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.256518 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.256522 | controller |     - '          exit 1'
2026-03-14 09:44:27.256527 | controller |     - '        fi'
2026-03-14 09:44:27.256531 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.256535 | controller |     - '        '
2026-03-14 09:44:27.256539 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.256543 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.256547 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.256552 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.256556 | controller |       --timeout=300s'
2026-03-14 09:44:27.256560 | controller |     - '        '
2026-03-14 09:44:27.256564 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.256568 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.256573 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.256577 | controller |     - '        '
2026-03-14 09:44:27.256581 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.256585 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.256589 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.256593 | controller |     - '        '
2026-03-14 09:44:27.256598 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.256602 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.256606 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.256610 | controller |       available"'
2026-03-14 09:44:27.256616 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.256621 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.256627 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.256633 | controller |     - '          exit 1'
2026-03-14 09:44:27.256639 | controller |     - '        fi'
2026-03-14 09:44:27.256645 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.256653 | controller |     - '        '
2026-03-14 09:44:27.256659 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.256672 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.256678 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.256683 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.256689 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.256693 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.256697 | controller |       -- \'
2026-03-14 09:44:27.256701 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.256706 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.256710 | controller |     - '    logger.go:42: 09:27:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.256714 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.256718 | controller |     - '    logger.go:42: 09:27:20 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.256722 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.256727 | controller |       found'
2026-03-14 09:44:27.256731 | controller |     - '    logger.go:42: 09:27:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.256735 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.256739 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.256743 | controller |     - '        '
2026-03-14 09:44:27.256748 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.256752 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.256756 | controller |     - '        '
2026-03-14 09:44:27.256760 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.256764 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.256768 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.256773 | controller |     - '          exit 1'
2026-03-14 09:44:27.256777 | controller |     - '        fi'
2026-03-14 09:44:27.256781 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.256785 | controller |     - '        '
2026-03-14 09:44:27.256789 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.256794 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.256798 | controller |       | base64 -d)'
2026-03-14 09:44:27.256802 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.256806 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.256810 | controller |     - '          exit 1'
2026-03-14 09:44:27.256815 | controller |     - '        fi'
2026-03-14 09:44:27.256819 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.256823 | controller |     - '        '
2026-03-14 09:44:27.256827 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.256831 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.256836 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.256840 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.256844 | controller |       --timeout=300s'
2026-03-14 09:44:27.256869 | controller |     - '        '
2026-03-14 09:44:27.256874 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.256881 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.256885 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.256889 | controller |     - '        '
2026-03-14 09:44:27.256893 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.256897 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.256902 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.256906 | controller |     - '        '
2026-03-14 09:44:27.256910 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.256915 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.256920 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.256925 | controller |       available"'
2026-03-14 09:44:27.256929 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.256934 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.256940 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.256944 | controller |     - '          exit 1'
2026-03-14 09:44:27.256948 | controller |     - '        fi'
2026-03-14 09:44:27.256953 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.256957 | controller |     - '        '
2026-03-14 09:44:27.256961 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.256971 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.256975 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.256980 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.256984 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.256988 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.256992 | controller |       -- \'
2026-03-14 09:44:27.256996 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.257001 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.257005 | controller |     - '    logger.go:42: 09:27:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.257009 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.257013 | controller |     - '    logger.go:42: 09:27:21 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.257017 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.257021 | controller |       found'
2026-03-14 09:44:27.257026 | controller |     - '    logger.go:42: 09:27:22 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.257030 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.257034 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.257038 | controller |     - '        '
2026-03-14 09:44:27.257043 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.257047 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.257051 | controller |     - '        '
2026-03-14 09:44:27.257055 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.257067 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.257071 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.257076 | controller |     - '          exit 1'
2026-03-14 09:44:27.257083 | controller |     - '        fi'
2026-03-14 09:44:27.257087 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.257091 | controller |     - '        '
2026-03-14 09:44:27.257095 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.257100 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.257104 | controller |       | base64 -d)'
2026-03-14 09:44:27.257108 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.257112 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.257116 | controller |     - '          exit 1'
2026-03-14 09:44:27.257120 | controller |     - '        fi'
2026-03-14 09:44:27.257125 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.257129 | controller |     - '        '
2026-03-14 09:44:27.257133 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.257137 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.257141 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.257145 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.257150 | controller |       --timeout=300s'
2026-03-14 09:44:27.257154 | controller |     - '        '
2026-03-14 09:44:27.257158 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.257162 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.257166 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.257171 | controller |     - '        '
2026-03-14 09:44:27.257175 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.257179 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.257183 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.257187 | controller |     - '        '
2026-03-14 09:44:27.257191 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.257196 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.257200 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.257204 | controller |       available"'
2026-03-14 09:44:27.257208 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.257214 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.257218 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.257222 | controller |     - '          exit 1'
2026-03-14 09:44:27.257226 | controller |     - '        fi'
2026-03-14 09:44:27.257231 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.257235 | controller |     - '        '
2026-03-14 09:44:27.257239 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.257243 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.257252 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.257257 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.257265 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.257269 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.257273 | controller |       -- \'
2026-03-14 09:44:27.257277 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.257281 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.257286 | controller |     - '    logger.go:42: 09:27:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.257304 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.257310 | controller |     - '    logger.go:42: 09:27:23 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.257315 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.257319 | controller |       found'
2026-03-14 09:44:27.257324 | controller |     - '    logger.go:42: 09:27:24 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.257328 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.257332 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.257336 | controller |     - '        '
2026-03-14 09:44:27.257341 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.257345 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.257349 | controller |     - '        '
2026-03-14 09:44:27.257353 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.257357 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.257362 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.257366 | controller |     - '          exit 1'
2026-03-14 09:44:27.257370 | controller |     - '        fi'
2026-03-14 09:44:27.257374 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.257378 | controller |     - '        '
2026-03-14 09:44:27.257382 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.257387 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.257391 | controller |       | base64 -d)'
2026-03-14 09:44:27.257395 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.257399 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.257403 | controller |     - '          exit 1'
2026-03-14 09:44:27.257407 | controller |     - '        fi'
2026-03-14 09:44:27.257412 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.257416 | controller |     - '        '
2026-03-14 09:44:27.257420 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.257424 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.257428 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.257432 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.257437 | controller |       --timeout=300s'
2026-03-14 09:44:27.257441 | controller |     - '        '
2026-03-14 09:44:27.257447 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.257452 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.257456 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.257460 | controller |     - '        '
2026-03-14 09:44:27.257464 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.257472 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.257476 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.257480 | controller |     - '        '
2026-03-14 09:44:27.257484 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.257489 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.257493 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.257497 | controller |       available"'
2026-03-14 09:44:27.257501 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.257505 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.257510 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.257516 | controller |     - '          exit 1'
2026-03-14 09:44:27.257520 | controller |     - '        fi'
2026-03-14 09:44:27.257524 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.257528 | controller |     - '        '
2026-03-14 09:44:27.257532 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.257537 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.257547 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.257552 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.257556 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.257560 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.257565 | controller |       -- \'
2026-03-14 09:44:27.257569 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.257573 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.257577 | controller |     - '    logger.go:42: 09:27:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.257581 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.257586 | controller |     - '    logger.go:42: 09:27:24 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.257590 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.257594 | controller |       found'
2026-03-14 09:44:27.257598 | controller |     - '    logger.go:42: 09:27:25 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.257602 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.257607 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.257611 | controller |     - '        '
2026-03-14 09:44:27.257616 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.257622 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.257628 | controller |     - '        '
2026-03-14 09:44:27.257636 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.257643 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.257648 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.257652 | controller |     - '          exit 1'
2026-03-14 09:44:27.257657 | controller |     - '        fi'
2026-03-14 09:44:27.257661 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.257665 | controller |     - '        '
2026-03-14 09:44:27.257669 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.257673 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.257680 | controller |       | base64 -d)'
2026-03-14 09:44:27.257684 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.257688 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.257692 | controller |     - '          exit 1'
2026-03-14 09:44:27.257697 | controller |     - '        fi'
2026-03-14 09:44:27.257701 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.257705 | controller |     - '        '
2026-03-14 09:44:27.257709 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.257713 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.257719 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.257724 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.257730 | controller |       --timeout=300s'
2026-03-14 09:44:27.257736 | controller |     - '        '
2026-03-14 09:44:27.257741 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.257746 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.257752 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.257758 | controller |     - '        '
2026-03-14 09:44:27.257764 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.257770 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.257776 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.257781 | controller |     - '        '
2026-03-14 09:44:27.257787 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.257793 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.257799 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.257805 | controller |       available"'
2026-03-14 09:44:27.257835 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.257841 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.257847 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.257852 | controller |     - '          exit 1'
2026-03-14 09:44:27.257858 | controller |     - '        fi'
2026-03-14 09:44:27.257863 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.257869 | controller |     - '        '
2026-03-14 09:44:27.257874 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.257880 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.257886 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.257900 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.257907 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.257914 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.257921 | controller |       -- \'
2026-03-14 09:44:27.257928 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.257932 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.257945 | controller |     - '    logger.go:42: 09:27:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.257950 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.257954 | controller |     - '    logger.go:42: 09:27:25 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.257959 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.257963 | controller |       found'
2026-03-14 09:44:27.257967 | controller |     - '    logger.go:42: 09:27:26 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.257971 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.257976 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.257980 | controller |     - '        '
2026-03-14 09:44:27.257984 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.257988 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.257993 | controller |     - '        '
2026-03-14 09:44:27.257997 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.258001 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.258005 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.258010 | controller |     - '          exit 1'
2026-03-14 09:44:27.258014 | controller |     - '        fi'
2026-03-14 09:44:27.258018 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.258022 | controller |     - '        '
2026-03-14 09:44:27.258027 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.258031 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.258035 | controller |       | base64 -d)'
2026-03-14 09:44:27.258039 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.258044 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.258048 | controller |     - '          exit 1'
2026-03-14 09:44:27.258052 | controller |     - '        fi'
2026-03-14 09:44:27.258056 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.258061 | controller |     - '        '
2026-03-14 09:44:27.258065 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.258069 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.258073 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.258077 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.258082 | controller |       --timeout=300s'
2026-03-14 09:44:27.258086 | controller |     - '        '
2026-03-14 09:44:27.258090 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.258094 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.258099 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.258103 | controller |     - '        '
2026-03-14 09:44:27.258107 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.258111 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.258115 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.258120 | controller |     - '        '
2026-03-14 09:44:27.258124 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.258128 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.258132 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.258139 | controller |       available"'
2026-03-14 09:44:27.258143 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.258147 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.258152 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.258156 | controller |     - '          exit 1'
2026-03-14 09:44:27.258160 | controller |     - '        fi'
2026-03-14 09:44:27.258164 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.258169 | controller |     - '        '
2026-03-14 09:44:27.258173 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.258177 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.258181 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.258191 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.258196 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.258201 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.258205 | controller |       -- \'
2026-03-14 09:44:27.258209 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.258213 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.258218 | controller |     - '    logger.go:42: 09:27:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.258222 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.258226 | controller |     - '    logger.go:42: 09:27:26 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.258230 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.258235 | controller |       found'
2026-03-14 09:44:27.258239 | controller |     - '    logger.go:42: 09:27:27 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.258243 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.258247 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.258251 | controller |     - '        '
2026-03-14 09:44:27.258255 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.258260 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.258264 | controller |     - '        '
2026-03-14 09:44:27.258268 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.258272 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.258276 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.258280 | controller |     - '          exit 1'
2026-03-14 09:44:27.258285 | controller |     - '        fi'
2026-03-14 09:44:27.258301 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.258319 | controller |     - '        '
2026-03-14 09:44:27.258323 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.258328 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.258332 | controller |       | base64 -d)'
2026-03-14 09:44:27.258336 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.258340 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.258344 | controller |     - '          exit 1'
2026-03-14 09:44:27.258349 | controller |     - '        fi'
2026-03-14 09:44:27.258353 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.258357 | controller |     - '        '
2026-03-14 09:44:27.258361 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.258368 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.258372 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.258376 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.258380 | controller |       --timeout=300s'
2026-03-14 09:44:27.258384 | controller |     - '        '
2026-03-14 09:44:27.258388 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.258393 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.258397 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.258401 | controller |     - '        '
2026-03-14 09:44:27.258405 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.258409 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.258414 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.258418 | controller |     - '        '
2026-03-14 09:44:27.258422 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.258426 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.258430 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.258435 | controller |       available"'
2026-03-14 09:44:27.258439 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.258443 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.258447 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.258451 | controller |     - '          exit 1'
2026-03-14 09:44:27.258456 | controller |     - '        fi'
2026-03-14 09:44:27.258460 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.258464 | controller |     - '        '
2026-03-14 09:44:27.258468 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.258472 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.258477 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.258481 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.258491 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.258495 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.258502 | controller |       -- \'
2026-03-14 09:44:27.258506 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.258510 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.258514 | controller |     - '    logger.go:42: 09:27:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.258518 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.258523 | controller |     - '    logger.go:42: 09:27:27 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.258527 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.258531 | controller |       found'
2026-03-14 09:44:27.258535 | controller |     - '    logger.go:42: 09:27:28 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.258542 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.258546 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.258550 | controller |     - '        '
2026-03-14 09:44:27.258554 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.258559 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.258563 | controller |     - '        '
2026-03-14 09:44:27.258567 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.258571 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.258575 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.258580 | controller |     - '          exit 1'
2026-03-14 09:44:27.258584 | controller |     - '        fi'
2026-03-14 09:44:27.258588 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.258592 | controller |     - '        '
2026-03-14 09:44:27.258605 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.258610 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.258615 | controller |       | base64 -d)'
2026-03-14 09:44:27.258623 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.258629 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.258635 | controller |     - '          exit 1'
2026-03-14 09:44:27.258641 | controller |     - '        fi'
2026-03-14 09:44:27.258647 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.258651 | controller |     - '        '
2026-03-14 09:44:27.258655 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.258660 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.258664 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.258668 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.258672 | controller |       --timeout=300s'
2026-03-14 09:44:27.258676 | controller |     - '        '
2026-03-14 09:44:27.258681 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.258685 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.258689 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.258693 | controller |     - '        '
2026-03-14 09:44:27.258697 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.258702 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.258706 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.258710 | controller |     - '        '
2026-03-14 09:44:27.258714 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.258718 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.258722 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.258727 | controller |       available"'
2026-03-14 09:44:27.258731 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.258735 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.258739 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.258743 | controller |     - '          exit 1'
2026-03-14 09:44:27.258747 | controller |     - '        fi'
2026-03-14 09:44:27.258752 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.258758 | controller |     - '        '
2026-03-14 09:44:27.258762 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.258766 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.258771 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.258775 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.258786 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.258791 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.258795 | controller |       -- \'
2026-03-14 09:44:27.258799 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.258803 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.258808 | controller |     - '    logger.go:42: 09:27:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.258812 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.258816 | controller |     - '    logger.go:42: 09:27:28 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.258820 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.258824 | controller |       found'
2026-03-14 09:44:27.258829 | controller |     - '    logger.go:42: 09:27:29 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.258833 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.258837 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.258841 | controller |     - '        '
2026-03-14 09:44:27.258846 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.258850 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.258854 | controller |     - '        '
2026-03-14 09:44:27.258858 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.258862 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.258866 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.258871 | controller |     - '          exit 1'
2026-03-14 09:44:27.258875 | controller |     - '        fi'
2026-03-14 09:44:27.258879 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.258883 | controller |     - '        '
2026-03-14 09:44:27.258888 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.258892 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.258896 | controller |       | base64 -d)'
2026-03-14 09:44:27.258900 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.258904 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.258909 | controller |     - '          exit 1'
2026-03-14 09:44:27.258913 | controller |     - '        fi'
2026-03-14 09:44:27.258917 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.258921 | controller |     - '        '
2026-03-14 09:44:27.258925 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.258930 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.258936 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.258941 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.258945 | controller |       --timeout=300s'
2026-03-14 09:44:27.258949 | controller |     - '        '
2026-03-14 09:44:27.258953 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.258959 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.258964 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.258968 | controller |     - '        '
2026-03-14 09:44:27.258972 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.258976 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.258980 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.258985 | controller |     - '        '
2026-03-14 09:44:27.258989 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.258993 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.258997 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.259001 | controller |       available"'
2026-03-14 09:44:27.259006 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.259010 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.259014 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.259018 | controller |     - '          exit 1'
2026-03-14 09:44:27.259022 | controller |     - '        fi'
2026-03-14 09:44:27.259027 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.259031 | controller |     - '        '
2026-03-14 09:44:27.259035 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.259039 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.259043 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.259048 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.259052 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.259061 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.259065 | controller |       -- \'
2026-03-14 09:44:27.259070 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.259074 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.259078 | controller |     - '    logger.go:42: 09:27:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.259082 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.259086 | controller |     - '    logger.go:42: 09:27:29 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.259091 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.259095 | controller |       found'
2026-03-14 09:44:27.259099 | controller |     - '    logger.go:42: 09:27:30 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.259103 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.259107 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.259112 | controller |     - '        '
2026-03-14 09:44:27.259116 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.259120 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.259124 | controller |     - '        '
2026-03-14 09:44:27.259128 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.259132 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.259145 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.259149 | controller |     - '          exit 1'
2026-03-14 09:44:27.259153 | controller |     - '        fi'
2026-03-14 09:44:27.259158 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.259162 | controller |     - '        '
2026-03-14 09:44:27.259166 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.259170 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.259174 | controller |       | base64 -d)'
2026-03-14 09:44:27.259179 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.259183 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.259187 | controller |     - '          exit 1'
2026-03-14 09:44:27.259191 | controller |     - '        fi'
2026-03-14 09:44:27.259195 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.259200 | controller |     - '        '
2026-03-14 09:44:27.259204 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.259208 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.259212 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.259216 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.259220 | controller |       --timeout=300s'
2026-03-14 09:44:27.259225 | controller |     - '        '
2026-03-14 09:44:27.259229 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.259233 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.259237 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.259241 | controller |     - '        '
2026-03-14 09:44:27.259248 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.259253 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.259257 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.259261 | controller |     - '        '
2026-03-14 09:44:27.259265 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.259269 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.259274 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.259278 | controller |       available"'
2026-03-14 09:44:27.259282 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.259289 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.259306 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.259311 | controller |     - '          exit 1'
2026-03-14 09:44:27.259315 | controller |     - '        fi'
2026-03-14 09:44:27.259319 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.259323 | controller |     - '        '
2026-03-14 09:44:27.259327 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.259332 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.259336 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.259340 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.259344 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.259357 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.259362 | controller |       -- \'
2026-03-14 09:44:27.259366 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.259370 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.259374 | controller |     - '    logger.go:42: 09:27:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.259379 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.259383 | controller |     - '    logger.go:42: 09:27:30 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.259387 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.259392 | controller |       found'
2026-03-14 09:44:27.259396 | controller |     - '    logger.go:42: 09:27:31 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.259400 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.259408 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.259413 | controller |     - '        '
2026-03-14 09:44:27.259417 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.259422 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.259426 | controller |     - '        '
2026-03-14 09:44:27.259430 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.259434 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.259439 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.259443 | controller |     - '          exit 1'
2026-03-14 09:44:27.259447 | controller |     - '        fi'
2026-03-14 09:44:27.259451 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.259456 | controller |     - '        '
2026-03-14 09:44:27.259460 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.259464 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.259468 | controller |       | base64 -d)'
2026-03-14 09:44:27.259472 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.259476 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.259481 | controller |     - '          exit 1'
2026-03-14 09:44:27.259485 | controller |     - '        fi'
2026-03-14 09:44:27.259489 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.259493 | controller |     - '        '
2026-03-14 09:44:27.259497 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.259502 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.259506 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.259520 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.259527 | controller |       --timeout=300s'
2026-03-14 09:44:27.259533 | controller |     - '        '
2026-03-14 09:44:27.259539 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.259545 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.259551 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.259557 | controller |     - '        '
2026-03-14 09:44:27.259563 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.259568 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.259585 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.259590 | controller |     - '        '
2026-03-14 09:44:27.259595 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.259599 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.259603 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.259607 | controller |       available"'
2026-03-14 09:44:27.259612 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.259618 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.259623 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.259629 | controller |     - '          exit 1'
2026-03-14 09:44:27.259635 | controller |     - '        fi'
2026-03-14 09:44:27.259641 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.259647 | controller |     - '        '
2026-03-14 09:44:27.259652 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.259657 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.259661 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.259665 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.259669 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.259673 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.259678 | controller |       -- \'
2026-03-14 09:44:27.259690 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.259696 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.259700 | controller |     - '    logger.go:42: 09:27:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.259704 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.259709 | controller |     - '    logger.go:42: 09:27:32 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.259713 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.259717 | controller |       found'
2026-03-14 09:44:27.259721 | controller |     - '    logger.go:42: 09:27:33 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.259725 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.259730 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.259734 | controller |     - '        '
2026-03-14 09:44:27.259748 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.259753 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.259757 | controller |     - '        '
2026-03-14 09:44:27.259762 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.259766 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.259770 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.259774 | controller |     - '          exit 1'
2026-03-14 09:44:27.259779 | controller |     - '        fi'
2026-03-14 09:44:27.259783 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.259787 | controller |     - '        '
2026-03-14 09:44:27.259791 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.259796 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.259803 | controller |       | base64 -d)'
2026-03-14 09:44:27.259807 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.259812 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.259816 | controller |     - '          exit 1'
2026-03-14 09:44:27.259820 | controller |     - '        fi'
2026-03-14 09:44:27.259824 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.259829 | controller |     - '        '
2026-03-14 09:44:27.259833 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.259837 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.259841 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.259846 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.259850 | controller |       --timeout=300s'
2026-03-14 09:44:27.259854 | controller |     - '        '
2026-03-14 09:44:27.259858 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.259862 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.259867 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.259871 | controller |     - '        '
2026-03-14 09:44:27.259875 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.259879 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.259883 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.259887 | controller |     - '        '
2026-03-14 09:44:27.259892 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.259896 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.259900 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.259904 | controller |       available"'
2026-03-14 09:44:27.259909 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.259915 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.259920 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.259925 | controller |     - '          exit 1'
2026-03-14 09:44:27.259930 | controller |     - '        fi'
2026-03-14 09:44:27.259934 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.259939 | controller |     - '        '
2026-03-14 09:44:27.259945 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.259949 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.259953 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.259957 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.259961 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.259965 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.259970 | controller |       -- \'
2026-03-14 09:44:27.259980 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.259984 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.259989 | controller |     - '    logger.go:42: 09:27:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.259998 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.260015 | controller |     - '    logger.go:42: 09:27:33 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.260025 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.260032 | controller |       found'
2026-03-14 09:44:27.260038 | controller |     - '    logger.go:42: 09:27:34 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.260044 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.260048 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.260052 | controller |     - '        '
2026-03-14 09:44:27.260056 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.260061 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.260065 | controller |     - '        '
2026-03-14 09:44:27.260069 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.260073 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.260078 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.260082 | controller |     - '          exit 1'
2026-03-14 09:44:27.260086 | controller |     - '        fi'
2026-03-14 09:44:27.260090 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.260094 | controller |     - '        '
2026-03-14 09:44:27.260099 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.260103 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.260107 | controller |       | base64 -d)'
2026-03-14 09:44:27.260111 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.260115 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.260119 | controller |     - '          exit 1'
2026-03-14 09:44:27.260124 | controller |     - '        fi'
2026-03-14 09:44:27.260128 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.260132 | controller |     - '        '
2026-03-14 09:44:27.260136 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.260140 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.260145 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.260149 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.260153 | controller |       --timeout=300s'
2026-03-14 09:44:27.260157 | controller |     - '        '
2026-03-14 09:44:27.260161 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.260166 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.260170 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.260174 | controller |     - '        '
2026-03-14 09:44:27.260178 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.260182 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.260187 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.260191 | controller |     - '        '
2026-03-14 09:44:27.260197 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.260201 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.260205 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.260210 | controller |       available"'
2026-03-14 09:44:27.260226 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.260231 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.260235 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.260239 | controller |     - '          exit 1'
2026-03-14 09:44:27.260243 | controller |     - '        fi'
2026-03-14 09:44:27.260247 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.260252 | controller |     - '        '
2026-03-14 09:44:27.260256 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.260260 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.260264 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.260270 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.260276 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.260282 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.260305 | controller |       -- \'
2026-03-14 09:44:27.260316 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.260328 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.260333 | controller |     - '    logger.go:42: 09:27:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.260337 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.260342 | controller |     - '    logger.go:42: 09:27:34 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.260346 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.260351 | controller |       found'
2026-03-14 09:44:27.260355 | controller |     - '    logger.go:42: 09:27:35 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.260359 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.260363 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.260367 | controller |     - '        '
2026-03-14 09:44:27.260371 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.260376 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.260380 | controller |     - '        '
2026-03-14 09:44:27.260384 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.260388 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.260392 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.260397 | controller |     - '          exit 1'
2026-03-14 09:44:27.260401 | controller |     - '        fi'
2026-03-14 09:44:27.260405 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.260409 | controller |     - '        '
2026-03-14 09:44:27.260413 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.260417 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.260422 | controller |       | base64 -d)'
2026-03-14 09:44:27.260426 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.260430 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.260434 | controller |     - '          exit 1'
2026-03-14 09:44:27.260439 | controller |     - '        fi'
2026-03-14 09:44:27.260443 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.260447 | controller |     - '        '
2026-03-14 09:44:27.260451 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.260455 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.260462 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.260466 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.260471 | controller |       --timeout=300s'
2026-03-14 09:44:27.260475 | controller |     - '        '
2026-03-14 09:44:27.260479 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.260483 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.260487 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.260491 | controller |     - '        '
2026-03-14 09:44:27.260496 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.260500 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.260504 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.260508 | controller |     - '        '
2026-03-14 09:44:27.260512 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.260517 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.260521 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.260525 | controller |       available"'
2026-03-14 09:44:27.260529 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.260533 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.260537 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.260542 | controller |     - '          exit 1'
2026-03-14 09:44:27.260546 | controller |     - '        fi'
2026-03-14 09:44:27.260550 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.260554 | controller |     - '        '
2026-03-14 09:44:27.260558 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.260562 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.260567 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.260571 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.260575 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.260579 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.260583 | controller |       -- \'
2026-03-14 09:44:27.260590 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.260599 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.260606 | controller |     - '    logger.go:42: 09:27:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.260610 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.260615 | controller |     - '    logger.go:42: 09:27:35 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.260621 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.260634 | controller |       found'
2026-03-14 09:44:27.260645 | controller |     - '    logger.go:42: 09:27:36 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.260651 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.260657 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.260666 | controller |     - '        '
2026-03-14 09:44:27.260671 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.260675 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.260679 | controller |     - '        '
2026-03-14 09:44:27.260683 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.260687 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.260692 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.260696 | controller |     - '          exit 1'
2026-03-14 09:44:27.260700 | controller |     - '        fi'
2026-03-14 09:44:27.260704 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.260708 | controller |     - '        '
2026-03-14 09:44:27.260713 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.260717 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.260721 | controller |       | base64 -d)'
2026-03-14 09:44:27.260725 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.260729 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.260736 | controller |     - '          exit 1'
2026-03-14 09:44:27.260741 | controller |     - '        fi'
2026-03-14 09:44:27.260745 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.260749 | controller |     - '        '
2026-03-14 09:44:27.260753 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.260757 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.260762 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.260766 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.260770 | controller |       --timeout=300s'
2026-03-14 09:44:27.260774 | controller |     - '        '
2026-03-14 09:44:27.260778 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.260782 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.260787 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.260791 | controller |     - '        '
2026-03-14 09:44:27.260795 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.260799 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.260803 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.260808 | controller |     - '        '
2026-03-14 09:44:27.260815 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.260819 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.260823 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.260829 | controller |       available"'
2026-03-14 09:44:27.260833 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.260837 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.260842 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.260846 | controller |     - '          exit 1'
2026-03-14 09:44:27.260850 | controller |     - '        fi'
2026-03-14 09:44:27.260854 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.260858 | controller |     - '        '
2026-03-14 09:44:27.260864 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.260868 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.260872 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.260877 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.260881 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.260885 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.260889 | controller |       -- \'
2026-03-14 09:44:27.260893 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.260897 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.260911 | controller |     - '    logger.go:42: 09:27:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.260916 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.260920 | controller |     - '    logger.go:42: 09:27:36 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.260925 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.260929 | controller |       found'
2026-03-14 09:44:27.260933 | controller |     - '    logger.go:42: 09:27:37 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.260938 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.260944 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.260948 | controller |     - '        '
2026-03-14 09:44:27.260952 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.260957 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.260963 | controller |     - '        '
2026-03-14 09:44:27.260967 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.260971 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.260975 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.260979 | controller |     - '          exit 1'
2026-03-14 09:44:27.260984 | controller |     - '        fi'
2026-03-14 09:44:27.260988 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.260992 | controller |     - '        '
2026-03-14 09:44:27.260996 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.261000 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.261004 | controller |       | base64 -d)'
2026-03-14 09:44:27.261009 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.261013 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.261017 | controller |     - '          exit 1'
2026-03-14 09:44:27.261021 | controller |     - '        fi'
2026-03-14 09:44:27.261025 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.261029 | controller |     - '        '
2026-03-14 09:44:27.261033 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.261038 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.261042 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.261046 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.261050 | controller |       --timeout=300s'
2026-03-14 09:44:27.261057 | controller |     - '        '
2026-03-14 09:44:27.261061 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.261065 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.261071 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.261075 | controller |     - '        '
2026-03-14 09:44:27.261079 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.261083 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.261087 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.261091 | controller |     - '        '
2026-03-14 09:44:27.261096 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.261100 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.261104 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.261108 | controller |       available"'
2026-03-14 09:44:27.261112 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.261116 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.261121 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.261125 | controller |     - '          exit 1'
2026-03-14 09:44:27.261130 | controller |     - '        fi'
2026-03-14 09:44:27.261135 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.261141 | controller |     - '        '
2026-03-14 09:44:27.261147 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.261152 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.261158 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.261163 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.261169 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.261175 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.261179 | controller |       -- \'
2026-03-14 09:44:27.261184 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.261188 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.261199 | controller |     - '    logger.go:42: 09:27:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.261204 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.261210 | controller |     - '    logger.go:42: 09:27:37 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.261216 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.261221 | controller |       found'
2026-03-14 09:44:27.261227 | controller |     - '    logger.go:42: 09:27:38 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.261234 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.261239 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.261245 | controller |     - '        '
2026-03-14 09:44:27.261250 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.261256 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.261261 | controller |     - '        '
2026-03-14 09:44:27.261265 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.261269 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.261273 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.261281 | controller |     - '          exit 1'
2026-03-14 09:44:27.261317 | controller |     - '        fi'
2026-03-14 09:44:27.261326 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.261332 | controller |     - '        '
2026-03-14 09:44:27.261341 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.261348 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.261354 | controller |       | base64 -d)'
2026-03-14 09:44:27.261360 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.261365 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.261371 | controller |     - '          exit 1'
2026-03-14 09:44:27.261376 | controller |     - '        fi'
2026-03-14 09:44:27.261382 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.261387 | controller |     - '        '
2026-03-14 09:44:27.261392 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.261398 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.261403 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.261409 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.261415 | controller |       --timeout=300s'
2026-03-14 09:44:27.261420 | controller |     - '        '
2026-03-14 09:44:27.261426 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.261431 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.261437 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.261443 | controller |     - '        '
2026-03-14 09:44:27.261448 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.261454 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.261459 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.261465 | controller |     - '        '
2026-03-14 09:44:27.261470 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.261476 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.261481 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.261487 | controller |       available"'
2026-03-14 09:44:27.261492 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.261501 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.261507 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.261513 | controller |     - '          exit 1'
2026-03-14 09:44:27.261519 | controller |     - '        fi'
2026-03-14 09:44:27.261524 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.261530 | controller |     - '        '
2026-03-14 09:44:27.261535 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.261541 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.261546 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.261552 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.261558 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.261566 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.261572 | controller |       -- \'
2026-03-14 09:44:27.261578 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.261584 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.261589 | controller |     - '    logger.go:42: 09:27:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.261606 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.261613 | controller |     - '    logger.go:42: 09:27:38 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.261619 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.261626 | controller |       found'
2026-03-14 09:44:27.261632 | controller |     - '    logger.go:42: 09:27:39 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.261638 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.261643 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.261649 | controller |     - '        '
2026-03-14 09:44:27.261654 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.261659 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.261664 | controller |     - '        '
2026-03-14 09:44:27.261670 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.261677 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.261682 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.261688 | controller |     - '          exit 1'
2026-03-14 09:44:27.261693 | controller |     - '        fi'
2026-03-14 09:44:27.261699 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.261704 | controller |     - '        '
2026-03-14 09:44:27.261710 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.261716 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.261721 | controller |       | base64 -d)'
2026-03-14 09:44:27.261727 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.261732 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.261738 | controller |     - '          exit 1'
2026-03-14 09:44:27.261743 | controller |     - '        fi'
2026-03-14 09:44:27.261749 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.261755 | controller |     - '        '
2026-03-14 09:44:27.261760 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.261766 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.261771 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.261777 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.261782 | controller |       --timeout=300s'
2026-03-14 09:44:27.261787 | controller |     - '        '
2026-03-14 09:44:27.261793 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.261798 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.261804 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.261809 | controller |     - '        '
2026-03-14 09:44:27.261841 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.261848 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.261854 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.261862 | controller |     - '        '
2026-03-14 09:44:27.261868 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.261877 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.261883 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.261888 | controller |       available"'
2026-03-14 09:44:27.261893 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.261899 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.261905 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.261910 | controller |     - '          exit 1'
2026-03-14 09:44:27.261916 | controller |     - '        fi'
2026-03-14 09:44:27.261921 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.261926 | controller |     - '        '
2026-03-14 09:44:27.261932 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.261937 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.261943 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.261949 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.261956 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.261961 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.261967 | controller |       -- \'
2026-03-14 09:44:27.261973 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.261978 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.261984 | controller |     - '    logger.go:42: 09:27:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.261990 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.262004 | controller |     - '    logger.go:42: 09:27:40 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.262011 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.262016 | controller |       found'
2026-03-14 09:44:27.262022 | controller |     - '    logger.go:42: 09:27:41 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.262027 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.262033 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.262039 | controller |     - '        '
2026-03-14 09:44:27.262045 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.262050 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.262056 | controller |     - '        '
2026-03-14 09:44:27.262062 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.262067 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.262073 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.262079 | controller |     - '          exit 1'
2026-03-14 09:44:27.262084 | controller |     - '        fi'
2026-03-14 09:44:27.262090 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.262096 | controller |     - '        '
2026-03-14 09:44:27.262101 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.262107 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.262113 | controller |       | base64 -d)'
2026-03-14 09:44:27.262119 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.262127 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.262133 | controller |     - '          exit 1'
2026-03-14 09:44:27.262139 | controller |     - '        fi'
2026-03-14 09:44:27.262147 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.262153 | controller |     - '        '
2026-03-14 09:44:27.262159 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.262164 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.262170 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.262176 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.262181 | controller |       --timeout=300s'
2026-03-14 09:44:27.262187 | controller |     - '        '
2026-03-14 09:44:27.262193 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.262199 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.262204 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.262210 | controller |     - '        '
2026-03-14 09:44:27.262216 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.262222 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.262228 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.262233 | controller |     - '        '
2026-03-14 09:44:27.262239 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.262244 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.262250 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.262256 | controller |       available"'
2026-03-14 09:44:27.262262 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.262267 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.262273 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.262279 | controller |     - '          exit 1'
2026-03-14 09:44:27.262284 | controller |     - '        fi'
2026-03-14 09:44:27.262306 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.262315 | controller |     - '        '
2026-03-14 09:44:27.262323 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.262329 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.262335 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.262341 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.262346 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.262352 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.262358 | controller |       -- \'
2026-03-14 09:44:27.262363 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.262369 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.262377 | controller |     - '    logger.go:42: 09:27:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.262387 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.262392 | controller |     - '    logger.go:42: 09:27:41 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.262406 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.262412 | controller |       found'
2026-03-14 09:44:27.262418 | controller |     - '    logger.go:42: 09:27:42 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.262424 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.262430 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.262436 | controller |     - '        '
2026-03-14 09:44:27.262441 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.262447 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.262453 | controller |     - '        '
2026-03-14 09:44:27.262459 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.262464 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.262470 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.262476 | controller |     - '          exit 1'
2026-03-14 09:44:27.262482 | controller |     - '        fi'
2026-03-14 09:44:27.262488 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.262494 | controller |     - '        '
2026-03-14 09:44:27.262499 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.262505 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.262511 | controller |       | base64 -d)'
2026-03-14 09:44:27.262517 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.262523 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.262529 | controller |     - '          exit 1'
2026-03-14 09:44:27.262534 | controller |     - '        fi'
2026-03-14 09:44:27.262540 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.262546 | controller |     - '        '
2026-03-14 09:44:27.262552 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.262557 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.262563 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.262569 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.262575 | controller |       --timeout=300s'
2026-03-14 09:44:27.262580 | controller |     - '        '
2026-03-14 09:44:27.262586 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.262592 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.262598 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.262604 | controller |     - '        '
2026-03-14 09:44:27.262609 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.262615 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.262621 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.262627 | controller |     - '        '
2026-03-14 09:44:27.262633 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.262639 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.262645 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.262651 | controller |       available"'
2026-03-14 09:44:27.262657 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.262666 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.262672 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.262678 | controller |     - '          exit 1'
2026-03-14 09:44:27.262684 | controller |     - '        fi'
2026-03-14 09:44:27.262690 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.262695 | controller |     - '        '
2026-03-14 09:44:27.262701 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.262707 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.262713 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.262719 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.262725 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.262731 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.262737 | controller |       -- \'
2026-03-14 09:44:27.262743 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.262749 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.262755 | controller |     - '    logger.go:42: 09:27:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.262761 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.262767 | controller |     - '    logger.go:42: 09:27:42 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.262780 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.262788 | controller |       found'
2026-03-14 09:44:27.262794 | controller |     - '    logger.go:42: 09:27:43 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.262800 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.262806 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.262812 | controller |     - '        '
2026-03-14 09:44:27.262818 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.262824 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.262830 | controller |     - '        '
2026-03-14 09:44:27.262836 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.262842 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.262848 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.262854 | controller |     - '          exit 1'
2026-03-14 09:44:27.262860 | controller |     - '        fi'
2026-03-14 09:44:27.262866 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.262872 | controller |     - '        '
2026-03-14 09:44:27.262882 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.262888 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.262894 | controller |       | base64 -d)'
2026-03-14 09:44:27.262900 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.262906 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.262917 | controller |     - '          exit 1'
2026-03-14 09:44:27.262924 | controller |     - '        fi'
2026-03-14 09:44:27.262929 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.262935 | controller |     - '        '
2026-03-14 09:44:27.262941 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.262947 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.262956 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.262962 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.262968 | controller |       --timeout=300s'
2026-03-14 09:44:27.262974 | controller |     - '        '
2026-03-14 09:44:27.262980 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.262986 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.262992 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.262999 | controller |     - '        '
2026-03-14 09:44:27.263005 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.263012 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.263018 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.263024 | controller |     - '        '
2026-03-14 09:44:27.263031 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.263038 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.263045 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.263052 | controller |       available"'
2026-03-14 09:44:27.263058 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.263068 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.263074 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.263080 | controller |     - '          exit 1'
2026-03-14 09:44:27.263086 | controller |     - '        fi'
2026-03-14 09:44:27.263092 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.263098 | controller |     - '        '
2026-03-14 09:44:27.263104 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.263110 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.263115 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.263137 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.263145 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.263152 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.263158 | controller |       -- \'
2026-03-14 09:44:27.263163 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.263168 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.263173 | controller |     - '    logger.go:42: 09:27:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.263177 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.263181 | controller |     - '    logger.go:42: 09:27:43 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.263186 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.263190 | controller |       found'
2026-03-14 09:44:27.263202 | controller |     - '    logger.go:42: 09:27:44 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.263206 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.263211 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.263215 | controller |     - '        '
2026-03-14 09:44:27.263222 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.263227 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.263233 | controller |     - '        '
2026-03-14 09:44:27.263239 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.263245 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.263251 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.263257 | controller |     - '          exit 1'
2026-03-14 09:44:27.263263 | controller |     - '        fi'
2026-03-14 09:44:27.263268 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.263273 | controller |     - '        '
2026-03-14 09:44:27.263279 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.263285 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.263312 | controller |       | base64 -d)'
2026-03-14 09:44:27.263317 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.263321 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.263325 | controller |     - '          exit 1'
2026-03-14 09:44:27.263330 | controller |     - '        fi'
2026-03-14 09:44:27.263334 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.263338 | controller |     - '        '
2026-03-14 09:44:27.263343 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.263347 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.263351 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.263356 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.263360 | controller |       --timeout=300s'
2026-03-14 09:44:27.263364 | controller |     - '        '
2026-03-14 09:44:27.263369 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.263373 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.263380 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.263384 | controller |     - '        '
2026-03-14 09:44:27.263389 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.263393 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.263397 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.263401 | controller |     - '        '
2026-03-14 09:44:27.263406 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.263410 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.263414 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.263419 | controller |       available"'
2026-03-14 09:44:27.263423 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.263427 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.263432 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.263436 | controller |     - '          exit 1'
2026-03-14 09:44:27.263440 | controller |     - '        fi'
2026-03-14 09:44:27.263444 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.263449 | controller |     - '        '
2026-03-14 09:44:27.263453 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.263459 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.263463 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.263468 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.263472 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.263476 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.263480 | controller |       -- \'
2026-03-14 09:44:27.263485 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.263489 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.263493 | controller |     - '    logger.go:42: 09:27:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.263497 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.263502 | controller |     - '    logger.go:42: 09:27:44 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.263506 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.263510 | controller |       found'
2026-03-14 09:44:27.263522 | controller |     - '    logger.go:42: 09:27:45 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.263527 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.263531 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.263535 | controller |     - '        '
2026-03-14 09:44:27.263540 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.263544 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.263548 | controller |     - '        '
2026-03-14 09:44:27.263552 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.263557 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.263563 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.263568 | controller |     - '          exit 1'
2026-03-14 09:44:27.263575 | controller |     - '        fi'
2026-03-14 09:44:27.263579 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.263583 | controller |     - '        '
2026-03-14 09:44:27.263588 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.263592 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.263596 | controller |       | base64 -d)'
2026-03-14 09:44:27.263601 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.263605 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.263609 | controller |     - '          exit 1'
2026-03-14 09:44:27.263614 | controller |     - '        fi'
2026-03-14 09:44:27.263618 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.263622 | controller |     - '        '
2026-03-14 09:44:27.263627 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.263631 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.263635 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.263640 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.263644 | controller |       --timeout=300s'
2026-03-14 09:44:27.263648 | controller |     - '        '
2026-03-14 09:44:27.263652 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.263657 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.263663 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.263667 | controller |     - '        '
2026-03-14 09:44:27.263672 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.263676 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.263680 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.263686 | controller |     - '        '
2026-03-14 09:44:27.263692 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.263698 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.263704 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.263709 | controller |       available"'
2026-03-14 09:44:27.263715 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.263720 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.263726 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.263731 | controller |     - '          exit 1'
2026-03-14 09:44:27.263737 | controller |     - '        fi'
2026-03-14 09:44:27.263743 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.263749 | controller |     - '        '
2026-03-14 09:44:27.263755 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.263760 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.263766 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.263772 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.263778 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.263784 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.263790 | controller |       -- \'
2026-03-14 09:44:27.263796 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.263801 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.263811 | controller |     - '    logger.go:42: 09:27:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.263817 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.263821 | controller |     - '    logger.go:42: 09:27:45 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.263825 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.263830 | controller |       found'
2026-03-14 09:44:27.263834 | controller |     - '    logger.go:42: 09:27:46 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.263838 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.263851 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.263855 | controller |     - '        '
2026-03-14 09:44:27.263860 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.263864 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.263868 | controller |     - '        '
2026-03-14 09:44:27.263872 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.263877 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.263881 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.263887 | controller |     - '          exit 1'
2026-03-14 09:44:27.263892 | controller |     - '        fi'
2026-03-14 09:44:27.263896 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.263900 | controller |     - '        '
2026-03-14 09:44:27.263904 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.263909 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.263913 | controller |       | base64 -d)'
2026-03-14 09:44:27.263918 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.263923 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.263928 | controller |     - '          exit 1'
2026-03-14 09:44:27.263932 | controller |     - '        fi'
2026-03-14 09:44:27.263937 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.263943 | controller |     - '        '
2026-03-14 09:44:27.263948 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.263952 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.263956 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.263961 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.263965 | controller |       --timeout=300s'
2026-03-14 09:44:27.263969 | controller |     - '        '
2026-03-14 09:44:27.263973 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.263978 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.263982 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.263986 | controller |     - '        '
2026-03-14 09:44:27.263993 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.263998 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.264002 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.264006 | controller |     - '        '
2026-03-14 09:44:27.264010 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.264015 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.264019 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.264023 | controller |       available"'
2026-03-14 09:44:27.264028 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.264032 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.264036 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.264040 | controller |     - '          exit 1'
2026-03-14 09:44:27.264045 | controller |     - '        fi'
2026-03-14 09:44:27.264049 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.264053 | controller |     - '        '
2026-03-14 09:44:27.264058 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.264062 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.264066 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.264070 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.264075 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.264079 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.264086 | controller |       -- \'
2026-03-14 09:44:27.264091 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.264095 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.264099 | controller |     - '    logger.go:42: 09:27:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.264104 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.264108 | controller |     - '    logger.go:42: 09:27:46 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.264112 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.264116 | controller |       found'
2026-03-14 09:44:27.264121 | controller |     - '    logger.go:42: 09:27:47 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.264125 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.264129 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.264133 | controller |     - '        '
2026-03-14 09:44:27.264144 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.264150 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.264157 | controller |     - '        '
2026-03-14 09:44:27.264163 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.264169 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.264175 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.264181 | controller |     - '          exit 1'
2026-03-14 09:44:27.264187 | controller |     - '        fi'
2026-03-14 09:44:27.264193 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.264198 | controller |     - '        '
2026-03-14 09:44:27.264204 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.264210 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.264217 | controller |       | base64 -d)'
2026-03-14 09:44:27.264223 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.264227 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.264232 | controller |     - '          exit 1'
2026-03-14 09:44:27.264236 | controller |     - '        fi'
2026-03-14 09:44:27.264240 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.264245 | controller |     - '        '
2026-03-14 09:44:27.264249 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.264253 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.264258 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.264263 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.264267 | controller |       --timeout=300s'
2026-03-14 09:44:27.264271 | controller |     - '        '
2026-03-14 09:44:27.264275 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.264280 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.264284 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.264304 | controller |     - '        '
2026-03-14 09:44:27.264311 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.264317 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.264321 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.264328 | controller |     - '        '
2026-03-14 09:44:27.264332 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.264337 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.264341 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.264345 | controller |       available"'
2026-03-14 09:44:27.264349 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.264354 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.264358 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.264362 | controller |     - '          exit 1'
2026-03-14 09:44:27.264367 | controller |     - '        fi'
2026-03-14 09:44:27.264371 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.264375 | controller |     - '        '
2026-03-14 09:44:27.264380 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.264384 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.264388 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.264392 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.264400 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.264404 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.264409 | controller |       -- \'
2026-03-14 09:44:27.264413 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.264417 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.264422 | controller |     - '    logger.go:42: 09:27:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.264426 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.264432 | controller |     - '    logger.go:42: 09:27:48 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.264436 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.264440 | controller |       found'
2026-03-14 09:44:27.264445 | controller |     - '    logger.go:42: 09:27:49 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.264449 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.264453 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.264458 | controller |     - '        '
2026-03-14 09:44:27.264470 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.264475 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.264479 | controller |     - '        '
2026-03-14 09:44:27.264483 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.264487 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.264492 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.264496 | controller |     - '          exit 1'
2026-03-14 09:44:27.264500 | controller |     - '        fi'
2026-03-14 09:44:27.264505 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.264509 | controller |     - '        '
2026-03-14 09:44:27.264515 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.264519 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.264524 | controller |       | base64 -d)'
2026-03-14 09:44:27.264528 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.264532 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.264538 | controller |     - '          exit 1'
2026-03-14 09:44:27.264543 | controller |     - '        fi'
2026-03-14 09:44:27.264547 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.264551 | controller |     - '        '
2026-03-14 09:44:27.264556 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.264560 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.264564 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.264568 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.264573 | controller |       --timeout=300s'
2026-03-14 09:44:27.264577 | controller |     - '        '
2026-03-14 09:44:27.264581 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.264586 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.264590 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.264594 | controller |     - '        '
2026-03-14 09:44:27.264598 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.264603 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.264607 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.264611 | controller |     - '        '
2026-03-14 09:44:27.264615 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.264620 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.264624 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.264628 | controller |       available"'
2026-03-14 09:44:27.264633 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.264637 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.264641 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.264646 | controller |     - '          exit 1'
2026-03-14 09:44:27.264652 | controller |     - '        fi'
2026-03-14 09:44:27.264658 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.264663 | controller |     - '        '
2026-03-14 09:44:27.264670 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.264676 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.264681 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.264687 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.264694 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.264700 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.264706 | controller |       -- \'
2026-03-14 09:44:27.264712 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.264717 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.264723 | controller |     - '    logger.go:42: 09:27:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.264729 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.264737 | controller |     - '    logger.go:42: 09:27:49 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.264743 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.264749 | controller |       found'
2026-03-14 09:44:27.264755 | controller |     - '    logger.go:42: 09:27:50 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.264759 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.264764 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.264768 | controller |     - '        '
2026-03-14 09:44:27.264772 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.264785 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.264789 | controller |     - '        '
2026-03-14 09:44:27.264793 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.264798 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.264802 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.264806 | controller |     - '          exit 1'
2026-03-14 09:44:27.264810 | controller |     - '        fi'
2026-03-14 09:44:27.264815 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.264819 | controller |     - '        '
2026-03-14 09:44:27.264823 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.264827 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.264832 | controller |       | base64 -d)'
2026-03-14 09:44:27.264836 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.264840 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.264844 | controller |     - '          exit 1'
2026-03-14 09:44:27.264848 | controller |     - '        fi'
2026-03-14 09:44:27.264853 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.264857 | controller |     - '        '
2026-03-14 09:44:27.264861 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.264865 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.264869 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.264874 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.264878 | controller |       --timeout=300s'
2026-03-14 09:44:27.264882 | controller |     - '        '
2026-03-14 09:44:27.264886 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.264890 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.264895 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.264899 | controller |     - '        '
2026-03-14 09:44:27.264903 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.264907 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.264911 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.264916 | controller |     - '        '
2026-03-14 09:44:27.264920 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.264924 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.264928 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.264932 | controller |       available"'
2026-03-14 09:44:27.264937 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.264941 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.264947 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.264951 | controller |     - '          exit 1'
2026-03-14 09:44:27.264955 | controller |     - '        fi'
2026-03-14 09:44:27.264960 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.264964 | controller |     - '        '
2026-03-14 09:44:27.264968 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.264972 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.264977 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.264981 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.264985 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.264989 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.264993 | controller |       -- \'
2026-03-14 09:44:27.264998 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.265002 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.265006 | controller |     - '    logger.go:42: 09:27:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.265010 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.265014 | controller |     - '    logger.go:42: 09:27:50 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.265021 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.265026 | controller |       found'
2026-03-14 09:44:27.265030 | controller |     - '    logger.go:42: 09:27:51 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.265035 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.265041 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.265047 | controller |     - '        '
2026-03-14 09:44:27.265053 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.265066 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.265073 | controller |     - '        '
2026-03-14 09:44:27.265079 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.265085 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.265091 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.265097 | controller |     - '          exit 1'
2026-03-14 09:44:27.265103 | controller |     - '        fi'
2026-03-14 09:44:27.265108 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.265114 | controller |     - '        '
2026-03-14 09:44:27.265120 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.265125 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.265131 | controller |       | base64 -d)'
2026-03-14 09:44:27.265136 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.265142 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.265147 | controller |     - '          exit 1'
2026-03-14 09:44:27.265153 | controller |     - '        fi'
2026-03-14 09:44:27.265159 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.265165 | controller |     - '        '
2026-03-14 09:44:27.265171 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.265177 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.265182 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.265191 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.265197 | controller |       --timeout=300s'
2026-03-14 09:44:27.265203 | controller |     - '        '
2026-03-14 09:44:27.265209 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.265214 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.265220 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.265226 | controller |     - '        '
2026-03-14 09:44:27.265231 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.265237 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.265242 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.265247 | controller |     - '        '
2026-03-14 09:44:27.265252 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.265258 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.265264 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.265270 | controller |       available"'
2026-03-14 09:44:27.265276 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.265282 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.265307 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.265317 | controller |     - '          exit 1'
2026-03-14 09:44:27.265322 | controller |     - '        fi'
2026-03-14 09:44:27.265326 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.265331 | controller |     - '        '
2026-03-14 09:44:27.265335 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.265339 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.265343 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.265348 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.265352 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.265356 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.265360 | controller |       -- \'
2026-03-14 09:44:27.265364 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.265368 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.265373 | controller |     - '    logger.go:42: 09:27:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.265377 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.265381 | controller |     - '    logger.go:42: 09:27:51 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.265385 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.265390 | controller |       found'
2026-03-14 09:44:27.265394 | controller |     - '    logger.go:42: 09:27:52 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.265398 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.265402 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.265406 | controller |     - '        '
2026-03-14 09:44:27.265411 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.265417 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.265422 | controller |     - '        '
2026-03-14 09:44:27.265433 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.265437 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.265442 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.265446 | controller |     - '          exit 1'
2026-03-14 09:44:27.265450 | controller |     - '        fi'
2026-03-14 09:44:27.265454 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.265459 | controller |     - '        '
2026-03-14 09:44:27.265463 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.265467 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.265471 | controller |       | base64 -d)'
2026-03-14 09:44:27.265475 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.265483 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.265487 | controller |     - '          exit 1'
2026-03-14 09:44:27.265491 | controller |     - '        fi'
2026-03-14 09:44:27.265496 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.265500 | controller |     - '        '
2026-03-14 09:44:27.265504 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.265508 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.265512 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.265517 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.265521 | controller |       --timeout=300s'
2026-03-14 09:44:27.265525 | controller |     - '        '
2026-03-14 09:44:27.265529 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.265535 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.265539 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.265544 | controller |     - '        '
2026-03-14 09:44:27.265548 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.265554 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.265558 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.265562 | controller |     - '        '
2026-03-14 09:44:27.265566 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.265571 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.265575 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.265579 | controller |       available"'
2026-03-14 09:44:27.265583 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.265587 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.265592 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.265596 | controller |     - '          exit 1'
2026-03-14 09:44:27.265600 | controller |     - '        fi'
2026-03-14 09:44:27.265604 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.265608 | controller |     - '        '
2026-03-14 09:44:27.265613 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.265617 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.265624 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.265628 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.265633 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.265637 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.265641 | controller |       -- \'
2026-03-14 09:44:27.265645 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.265649 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.265654 | controller |     - '    logger.go:42: 09:27:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.265658 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.265662 | controller |     - '    logger.go:42: 09:27:52 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.265666 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.265671 | controller |       found'
2026-03-14 09:44:27.265676 | controller |     - '    logger.go:42: 09:27:53 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.265680 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.265685 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.265689 | controller |     - '        '
2026-03-14 09:44:27.265693 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.265697 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.265702 | controller |     - '        '
2026-03-14 09:44:27.265712 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.265716 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.265721 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.265725 | controller |     - '          exit 1'
2026-03-14 09:44:27.265729 | controller |     - '        fi'
2026-03-14 09:44:27.265733 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.265738 | controller |     - '        '
2026-03-14 09:44:27.265742 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.265746 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.265750 | controller |       | base64 -d)'
2026-03-14 09:44:27.265755 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.265759 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.265763 | controller |     - '          exit 1'
2026-03-14 09:44:27.265767 | controller |     - '        fi'
2026-03-14 09:44:27.265772 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.265776 | controller |     - '        '
2026-03-14 09:44:27.265780 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.265784 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.265789 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.265793 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.265797 | controller |       --timeout=300s'
2026-03-14 09:44:27.265804 | controller |     - '        '
2026-03-14 09:44:27.265808 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.265812 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.265817 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.265823 | controller |     - '        '
2026-03-14 09:44:27.265846 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.265851 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.265855 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.265859 | controller |     - '        '
2026-03-14 09:44:27.265863 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.265868 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.265872 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.265876 | controller |       available"'
2026-03-14 09:44:27.265880 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.265885 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.265889 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.265893 | controller |     - '          exit 1'
2026-03-14 09:44:27.265899 | controller |     - '        fi'
2026-03-14 09:44:27.265903 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.265908 | controller |     - '        '
2026-03-14 09:44:27.265912 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.265916 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.265921 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.265927 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.265931 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.265936 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.265941 | controller |       -- \'
2026-03-14 09:44:27.265946 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.265950 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.265954 | controller |     - '    logger.go:42: 09:27:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.265959 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.265963 | controller |     - '    logger.go:42: 09:27:53 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.265967 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.265971 | controller |       found'
2026-03-14 09:44:27.265975 | controller |     - '    logger.go:42: 09:27:54 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.265980 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.265984 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.265988 | controller |     - '        '
2026-03-14 09:44:27.265992 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.265997 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.266001 | controller |     - '        '
2026-03-14 09:44:27.266005 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.266016 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.266020 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.266024 | controller |     - '          exit 1'
2026-03-14 09:44:27.266029 | controller |     - '        fi'
2026-03-14 09:44:27.266035 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.266039 | controller |     - '        '
2026-03-14 09:44:27.266043 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.266048 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.266052 | controller |       | base64 -d)'
2026-03-14 09:44:27.266056 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.266060 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.266064 | controller |     - '          exit 1'
2026-03-14 09:44:27.266069 | controller |     - '        fi'
2026-03-14 09:44:27.266073 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.266077 | controller |     - '        '
2026-03-14 09:44:27.266081 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.266085 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.266090 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.266094 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.266098 | controller |       --timeout=300s'
2026-03-14 09:44:27.266102 | controller |     - '        '
2026-03-14 09:44:27.266106 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.266110 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.266115 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.266119 | controller |     - '        '
2026-03-14 09:44:27.266123 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.266127 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.266131 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.266136 | controller |     - '        '
2026-03-14 09:44:27.266140 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.266144 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.266148 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.266152 | controller |       available"'
2026-03-14 09:44:27.266157 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.266161 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.266165 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.266169 | controller |     - '          exit 1'
2026-03-14 09:44:27.266173 | controller |     - '        fi'
2026-03-14 09:44:27.266178 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.266182 | controller |     - '        '
2026-03-14 09:44:27.266186 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.266190 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.266194 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.266198 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.266203 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.266207 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.266213 | controller |       -- \'
2026-03-14 09:44:27.266217 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.266221 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.266225 | controller |     - '    logger.go:42: 09:27:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.266230 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.266234 | controller |     - '    logger.go:42: 09:27:55 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.266238 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.266242 | controller |       found'
2026-03-14 09:44:27.266246 | controller |     - '    logger.go:42: 09:27:56 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.266251 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.266255 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.266259 | controller |     - '        '
2026-03-14 09:44:27.266265 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.266269 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.266273 | controller |     - '        '
2026-03-14 09:44:27.266278 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.266282 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.266304 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.266312 | controller |     - '          exit 1'
2026-03-14 09:44:27.266316 | controller |     - '        fi'
2026-03-14 09:44:27.266320 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.266325 | controller |     - '        '
2026-03-14 09:44:27.266329 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.266335 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.266340 | controller |       | base64 -d)'
2026-03-14 09:44:27.266346 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.266352 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.266358 | controller |     - '          exit 1'
2026-03-14 09:44:27.266364 | controller |     - '        fi'
2026-03-14 09:44:27.266370 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.266376 | controller |     - '        '
2026-03-14 09:44:27.266381 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.266387 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.266392 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.266398 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.266403 | controller |       --timeout=300s'
2026-03-14 09:44:27.266408 | controller |     - '        '
2026-03-14 09:44:27.266414 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.266420 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.266426 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.266431 | controller |     - '        '
2026-03-14 09:44:27.266437 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.266443 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.266448 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.266454 | controller |     - '        '
2026-03-14 09:44:27.266463 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.266469 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.266475 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.266481 | controller |       available"'
2026-03-14 09:44:27.266487 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.266493 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.266498 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.266504 | controller |     - '          exit 1'
2026-03-14 09:44:27.266510 | controller |     - '        fi'
2026-03-14 09:44:27.266516 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.266521 | controller |     - '        '
2026-03-14 09:44:27.266527 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.266534 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.266539 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.266545 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.266551 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.266557 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.266566 | controller |       -- \'
2026-03-14 09:44:27.266572 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.266576 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.266580 | controller |     - '    logger.go:42: 09:27:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.266585 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.266589 | controller |     - '    logger.go:42: 09:27:56 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.266593 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.266598 | controller |       found'
2026-03-14 09:44:27.266604 | controller |     - '    logger.go:42: 09:27:57 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.266610 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.266616 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.266621 | controller |     - '        '
2026-03-14 09:44:27.266627 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.266633 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.266639 | controller |     - '        '
2026-03-14 09:44:27.266645 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.266651 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.266657 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.266674 | controller |     - '          exit 1'
2026-03-14 09:44:27.266681 | controller |     - '        fi'
2026-03-14 09:44:27.266687 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.266692 | controller |     - '        '
2026-03-14 09:44:27.266697 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.266701 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.266705 | controller |       | base64 -d)'
2026-03-14 09:44:27.266709 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.266713 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.266720 | controller |     - '          exit 1'
2026-03-14 09:44:27.266725 | controller |     - '        fi'
2026-03-14 09:44:27.266729 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.266733 | controller |     - '        '
2026-03-14 09:44:27.266737 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.266742 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.266746 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.266750 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.266754 | controller |       --timeout=300s'
2026-03-14 09:44:27.266758 | controller |     - '        '
2026-03-14 09:44:27.266763 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.266767 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.266771 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.266775 | controller |     - '        '
2026-03-14 09:44:27.266779 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.266784 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.266788 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.266792 | controller |     - '        '
2026-03-14 09:44:27.266796 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.266800 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.266805 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.266810 | controller |       available"'
2026-03-14 09:44:27.266816 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.266822 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.266828 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.266834 | controller |     - '          exit 1'
2026-03-14 09:44:27.266840 | controller |     - '        fi'
2026-03-14 09:44:27.266846 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.266852 | controller |     - '        '
2026-03-14 09:44:27.266858 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.266863 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.266869 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.266875 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.266881 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.266887 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.266893 | controller |       -- \'
2026-03-14 09:44:27.266899 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.266904 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.266914 | controller |     - '    logger.go:42: 09:27:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.266922 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.266928 | controller |     - '    logger.go:42: 09:27:57 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.266937 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.266944 | controller |       found'
2026-03-14 09:44:27.266950 | controller |     - '    logger.go:42: 09:27:58 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.266956 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.266961 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.266967 | controller |     - '        '
2026-03-14 09:44:27.266973 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.266981 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.266988 | controller |     - '        '
2026-03-14 09:44:27.266993 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.266999 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.267005 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.267010 | controller |     - '          exit 1'
2026-03-14 09:44:27.267016 | controller |     - '        fi'
2026-03-14 09:44:27.267035 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.267042 | controller |     - '        '
2026-03-14 09:44:27.267048 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.267054 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.267060 | controller |       | base64 -d)'
2026-03-14 09:44:27.267065 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.267071 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.267077 | controller |     - '          exit 1'
2026-03-14 09:44:27.267083 | controller |     - '        fi'
2026-03-14 09:44:27.267089 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.267095 | controller |     - '        '
2026-03-14 09:44:27.267100 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.267106 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.267112 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.267118 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.267124 | controller |       --timeout=300s'
2026-03-14 09:44:27.267130 | controller |     - '        '
2026-03-14 09:44:27.267136 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.267140 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.267144 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.267149 | controller |     - '        '
2026-03-14 09:44:27.267153 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.267157 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.267161 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.267165 | controller |     - '        '
2026-03-14 09:44:27.267170 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.267174 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.267178 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.267182 | controller |       available"'
2026-03-14 09:44:27.267187 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.267191 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.267198 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.267202 | controller |     - '          exit 1'
2026-03-14 09:44:27.267206 | controller |     - '        fi'
2026-03-14 09:44:27.267210 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.267215 | controller |     - '        '
2026-03-14 09:44:27.267219 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.267223 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.267227 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.267231 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.267236 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.267240 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.267244 | controller |       -- \'
2026-03-14 09:44:27.267248 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.267252 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.267257 | controller |     - '    logger.go:42: 09:27:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.267261 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.267265 | controller |     - '    logger.go:42: 09:27:58 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.267269 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.267273 | controller |       found'
2026-03-14 09:44:27.267278 | controller |     - '    logger.go:42: 09:27:59 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.267282 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.267301 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.267308 | controller |     - '        '
2026-03-14 09:44:27.267313 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.267317 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.267321 | controller |     - '        '
2026-03-14 09:44:27.267325 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.267330 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.267334 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.267338 | controller |     - '          exit 1'
2026-03-14 09:44:27.267342 | controller |     - '        fi'
2026-03-14 09:44:27.267347 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.267351 | controller |     - '        '
2026-03-14 09:44:27.267365 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.267370 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.267374 | controller |       | base64 -d)'
2026-03-14 09:44:27.267378 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.267382 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.267387 | controller |     - '          exit 1'
2026-03-14 09:44:27.267391 | controller |     - '        fi'
2026-03-14 09:44:27.267395 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.267399 | controller |     - '        '
2026-03-14 09:44:27.267403 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.267408 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.267412 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.267418 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.267422 | controller |       --timeout=300s'
2026-03-14 09:44:27.267426 | controller |     - '        '
2026-03-14 09:44:27.267431 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.267435 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.267439 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.267443 | controller |     - '        '
2026-03-14 09:44:27.267447 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.267452 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.267456 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.267460 | controller |     - '        '
2026-03-14 09:44:27.267464 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.267468 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.267473 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.267477 | controller |       available"'
2026-03-14 09:44:27.267481 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.267485 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.267489 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.267494 | controller |     - '          exit 1'
2026-03-14 09:44:27.267498 | controller |     - '        fi'
2026-03-14 09:44:27.267502 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.267506 | controller |     - '        '
2026-03-14 09:44:27.267510 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.267515 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.267519 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.267523 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.267527 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.267531 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.267536 | controller |       -- \'
2026-03-14 09:44:27.267540 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.267544 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.267548 | controller |     - '    logger.go:42: 09:27:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.267552 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.267557 | controller |     - '    logger.go:42: 09:27:59 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.267561 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.267565 | controller |       found'
2026-03-14 09:44:27.267569 | controller |     - '    logger.go:42: 09:28:00 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.267573 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.267579 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.267584 | controller |     - '        '
2026-03-14 09:44:27.267588 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.267594 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.267598 | controller |     - '        '
2026-03-14 09:44:27.267604 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.267609 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.267613 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.267618 | controller |     - '          exit 1'
2026-03-14 09:44:27.267622 | controller |     - '        fi'
2026-03-14 09:44:27.267627 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.267633 | controller |     - '        '
2026-03-14 09:44:27.267646 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.267653 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.267658 | controller |       | base64 -d)'
2026-03-14 09:44:27.267664 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.267670 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.267676 | controller |     - '          exit 1'
2026-03-14 09:44:27.267682 | controller |     - '        fi'
2026-03-14 09:44:27.267688 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.267693 | controller |     - '        '
2026-03-14 09:44:27.267699 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.267705 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.267711 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.267717 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.267722 | controller |       --timeout=300s'
2026-03-14 09:44:27.267728 | controller |     - '        '
2026-03-14 09:44:27.267733 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.267739 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.267745 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.267752 | controller |     - '        '
2026-03-14 09:44:27.267757 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.267763 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.267772 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.267778 | controller |     - '        '
2026-03-14 09:44:27.267784 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.267790 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.267796 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.267800 | controller |       available"'
2026-03-14 09:44:27.267804 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.267809 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.267813 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.267817 | controller |     - '          exit 1'
2026-03-14 09:44:27.267821 | controller |     - '        fi'
2026-03-14 09:44:27.267826 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.267830 | controller |     - '        '
2026-03-14 09:44:27.267834 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.267838 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.267845 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.267849 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.267853 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.267858 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.267862 | controller |       -- \'
2026-03-14 09:44:27.267866 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.267871 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.267875 | controller |     - '    logger.go:42: 09:28:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.267879 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.267883 | controller |     - '    logger.go:42: 09:28:00 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.267888 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.267892 | controller |       found'
2026-03-14 09:44:27.267896 | controller |     - '    logger.go:42: 09:28:01 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.267900 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.267905 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.267909 | controller |     - '        '
2026-03-14 09:44:27.267913 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.267918 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.267925 | controller |     - '        '
2026-03-14 09:44:27.267942 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.267949 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.267955 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.267962 | controller |     - '          exit 1'
2026-03-14 09:44:27.267968 | controller |     - '        fi'
2026-03-14 09:44:27.267973 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.267979 | controller |     - '        '
2026-03-14 09:44:27.267985 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.268000 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.268007 | controller |       | base64 -d)'
2026-03-14 09:44:27.268013 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.268018 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.268024 | controller |     - '          exit 1'
2026-03-14 09:44:27.268030 | controller |     - '        fi'
2026-03-14 09:44:27.268036 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.268042 | controller |     - '        '
2026-03-14 09:44:27.268048 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.268054 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.268060 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.268065 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.268071 | controller |       --timeout=300s'
2026-03-14 09:44:27.268077 | controller |     - '        '
2026-03-14 09:44:27.268083 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.268089 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.268094 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.268104 | controller |     - '        '
2026-03-14 09:44:27.268110 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.268116 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.268121 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.268128 | controller |     - '        '
2026-03-14 09:44:27.268133 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.268139 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.268145 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.268151 | controller |       available"'
2026-03-14 09:44:27.268156 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.268162 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.268168 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.268174 | controller |     - '          exit 1'
2026-03-14 09:44:27.268179 | controller |     - '        fi'
2026-03-14 09:44:27.268189 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.268195 | controller |     - '        '
2026-03-14 09:44:27.268203 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.268209 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.268215 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.268221 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.268227 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.268233 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.268239 | controller |       -- \'
2026-03-14 09:44:27.268244 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.268250 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.268256 | controller |     - '    logger.go:42: 09:28:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.268262 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.268268 | controller |     - '    logger.go:42: 09:28:02 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.268273 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.268279 | controller |       found'
2026-03-14 09:44:27.268285 | controller |     - '    logger.go:42: 09:28:03 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.268312 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.268319 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.268325 | controller |     - '        '
2026-03-14 09:44:27.268331 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.268336 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.268342 | controller |     - '        '
2026-03-14 09:44:27.268348 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.268354 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.268360 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.268366 | controller |     - '          exit 1'
2026-03-14 09:44:27.268372 | controller |     - '        fi'
2026-03-14 09:44:27.268377 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.268389 | controller |     - '        '
2026-03-14 09:44:27.268395 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.268410 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.268416 | controller |       | base64 -d)'
2026-03-14 09:44:27.268422 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.268428 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.268434 | controller |     - '          exit 1'
2026-03-14 09:44:27.268440 | controller |     - '        fi'
2026-03-14 09:44:27.268445 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.268451 | controller |     - '        '
2026-03-14 09:44:27.268457 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.268463 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.268469 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.268475 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.268481 | controller |       --timeout=300s'
2026-03-14 09:44:27.268487 | controller |     - '        '
2026-03-14 09:44:27.268492 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.268498 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.268504 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.268510 | controller |     - '        '
2026-03-14 09:44:27.268518 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.268524 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.268530 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.268536 | controller |     - '        '
2026-03-14 09:44:27.268542 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.268548 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.268554 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.268560 | controller |       available"'
2026-03-14 09:44:27.268565 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.268571 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.268577 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.268583 | controller |     - '          exit 1'
2026-03-14 09:44:27.268588 | controller |     - '        fi'
2026-03-14 09:44:27.268594 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.268600 | controller |     - '        '
2026-03-14 09:44:27.268606 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.268611 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.268617 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.268623 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.268629 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.268635 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.268641 | controller |       -- \'
2026-03-14 09:44:27.268646 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.268657 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.268663 | controller |     - '    logger.go:42: 09:28:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.268669 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.268675 | controller |     - '    logger.go:42: 09:28:03 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.268680 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.268686 | controller |       found'
2026-03-14 09:44:27.268692 | controller |     - '    logger.go:42: 09:28:04 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.268698 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.268704 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.268709 | controller |     - '        '
2026-03-14 09:44:27.268716 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.268721 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.268727 | controller |     - '        '
2026-03-14 09:44:27.268733 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.268739 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.268745 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.268750 | controller |     - '          exit 1'
2026-03-14 09:44:27.268756 | controller |     - '        fi'
2026-03-14 09:44:27.268762 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.268768 | controller |     - '        '
2026-03-14 09:44:27.268774 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.268779 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.268785 | controller |       | base64 -d)'
2026-03-14 09:44:27.268800 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.268807 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.268813 | controller |     - '          exit 1'
2026-03-14 09:44:27.268819 | controller |     - '        fi'
2026-03-14 09:44:27.268825 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.268830 | controller |     - '        '
2026-03-14 09:44:27.268836 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.268842 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.268848 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.268854 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.268859 | controller |       --timeout=300s'
2026-03-14 09:44:27.268865 | controller |     - '        '
2026-03-14 09:44:27.268871 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.268877 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.268883 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.268889 | controller |     - '        '
2026-03-14 09:44:27.268895 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.268900 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.268906 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.268912 | controller |     - '        '
2026-03-14 09:44:27.268918 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.268927 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.268933 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.268939 | controller |       available"'
2026-03-14 09:44:27.268944 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.268953 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.268959 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.268965 | controller |     - '          exit 1'
2026-03-14 09:44:27.268971 | controller |     - '        fi'
2026-03-14 09:44:27.268977 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.268983 | controller |     - '        '
2026-03-14 09:44:27.268989 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.268995 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.269000 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.269006 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.269012 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.269018 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.269024 | controller |       -- \'
2026-03-14 09:44:27.269029 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.269035 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.269044 | controller |     - '    logger.go:42: 09:28:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.269050 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.269056 | controller |     - '    logger.go:42: 09:28:04 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.269062 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.269068 | controller |       found'
2026-03-14 09:44:27.269073 | controller |     - '    logger.go:42: 09:28:05 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.269079 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.269085 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.269091 | controller |     - '        '
2026-03-14 09:44:27.269097 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.269102 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.269108 | controller |     - '        '
2026-03-14 09:44:27.269114 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.269120 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.269126 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.269132 | controller |     - '          exit 1'
2026-03-14 09:44:27.269137 | controller |     - '        fi'
2026-03-14 09:44:27.269143 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.269149 | controller |     - '        '
2026-03-14 09:44:27.269155 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.269161 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.269166 | controller |       | base64 -d)'
2026-03-14 09:44:27.269172 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.269186 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.269193 | controller |     - '          exit 1'
2026-03-14 09:44:27.269201 | controller |     - '        fi'
2026-03-14 09:44:27.269208 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.269214 | controller |     - '        '
2026-03-14 09:44:27.269220 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.269226 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.269231 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.269240 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.269246 | controller |       --timeout=300s'
2026-03-14 09:44:27.269252 | controller |     - '        '
2026-03-14 09:44:27.269258 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.269263 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.269269 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.269275 | controller |     - '        '
2026-03-14 09:44:27.269281 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.269290 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.269312 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.269319 | controller |     - '        '
2026-03-14 09:44:27.269324 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.269330 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.269336 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.269342 | controller |       available"'
2026-03-14 09:44:27.269348 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.269353 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.269359 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.269365 | controller |     - '          exit 1'
2026-03-14 09:44:27.269371 | controller |     - '        fi'
2026-03-14 09:44:27.269377 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.269383 | controller |     - '        '
2026-03-14 09:44:27.269389 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.269395 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.269401 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.269406 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.269412 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.269418 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.269424 | controller |       -- \'
2026-03-14 09:44:27.269429 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.269435 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.269441 | controller |     - '    logger.go:42: 09:28:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.269447 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.269453 | controller |     - '    logger.go:42: 09:28:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.269459 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.269468 | controller |       found'
2026-03-14 09:44:27.269474 | controller |     - '    logger.go:42: 09:28:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.269480 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.269486 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.269492 | controller |     - '        '
2026-03-14 09:44:27.269497 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.269503 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.269509 | controller |     - '        '
2026-03-14 09:44:27.269515 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.269520 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.269526 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.269532 | controller |     - '          exit 1'
2026-03-14 09:44:27.269538 | controller |     - '        fi'
2026-03-14 09:44:27.269544 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.269550 | controller |     - '        '
2026-03-14 09:44:27.269556 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.269561 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.269567 | controller |       | base64 -d)'
2026-03-14 09:44:27.269573 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.269588 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.269594 | controller |     - '          exit 1'
2026-03-14 09:44:27.269600 | controller |     - '        fi'
2026-03-14 09:44:27.269605 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.269611 | controller |     - '        '
2026-03-14 09:44:27.269617 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.269623 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.269629 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.269635 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.269641 | controller |       --timeout=300s'
2026-03-14 09:44:27.269646 | controller |     - '        '
2026-03-14 09:44:27.269652 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.269658 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.269664 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.269670 | controller |     - '        '
2026-03-14 09:44:27.269679 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.269685 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.269690 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.269696 | controller |     - '        '
2026-03-14 09:44:27.269702 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.269708 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.269713 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.269718 | controller |       available"'
2026-03-14 09:44:27.269723 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.269729 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.269734 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.269741 | controller |     - '          exit 1'
2026-03-14 09:44:27.269747 | controller |     - '        fi'
2026-03-14 09:44:27.269752 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.269758 | controller |     - '        '
2026-03-14 09:44:27.269764 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.269770 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.269776 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.269782 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.269787 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.269793 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.269799 | controller |       -- \'
2026-03-14 09:44:27.269805 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.269810 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.269816 | controller |     - '    logger.go:42: 09:28:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.269822 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.269828 | controller |     - '    logger.go:42: 09:28:06 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.269850 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.269857 | controller |       found'
2026-03-14 09:44:27.269862 | controller |     - '    logger.go:42: 09:28:07 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.269868 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.269874 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.269880 | controller |     - '        '
2026-03-14 09:44:27.269886 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.269892 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.269897 | controller |     - '        '
2026-03-14 09:44:27.269903 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.269909 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.269915 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.269921 | controller |     - '          exit 1'
2026-03-14 09:44:27.269926 | controller |     - '        fi'
2026-03-14 09:44:27.269932 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.269938 | controller |     - '        '
2026-03-14 09:44:27.269944 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.269950 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.269956 | controller |       | base64 -d)'
2026-03-14 09:44:27.269962 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.269968 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.269974 | controller |     - '          exit 1'
2026-03-14 09:44:27.269988 | controller |     - '        fi'
2026-03-14 09:44:27.269994 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.270000 | controller |     - '        '
2026-03-14 09:44:27.270006 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.270011 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.270017 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.270055 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.270073 | controller |       --timeout=300s'
2026-03-14 09:44:27.270079 | controller |     - '        '
2026-03-14 09:44:27.270085 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.270090 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.270096 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.270102 | controller |     - '        '
2026-03-14 09:44:27.270108 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.270114 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.270120 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.270126 | controller |     - '        '
2026-03-14 09:44:27.270132 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.270137 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.270143 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.270149 | controller |       available"'
2026-03-14 09:44:27.270155 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.270161 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.270167 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.270172 | controller |     - '          exit 1'
2026-03-14 09:44:27.270178 | controller |     - '        fi'
2026-03-14 09:44:27.270184 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.270190 | controller |     - '        '
2026-03-14 09:44:27.270195 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.270201 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.270207 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.270213 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.270219 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.270225 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.270231 | controller |       -- \'
2026-03-14 09:44:27.270237 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.270243 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.270248 | controller |     - '    logger.go:42: 09:28:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.270254 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.270260 | controller |     - '    logger.go:42: 09:28:07 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.270266 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.270272 | controller |       found'
2026-03-14 09:44:27.270277 | controller |     - '    logger.go:42: 09:28:08 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.270283 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.270304 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.270314 | controller |     - '        '
2026-03-14 09:44:27.270319 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.270325 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.270334 | controller |     - '        '
2026-03-14 09:44:27.270340 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.270346 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.270352 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.270358 | controller |     - '          exit 1'
2026-03-14 09:44:27.270364 | controller |     - '        fi'
2026-03-14 09:44:27.270369 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.270375 | controller |     - '        '
2026-03-14 09:44:27.270381 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.270387 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.270393 | controller |       | base64 -d)'
2026-03-14 09:44:27.270399 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.270416 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.270423 | controller |     - '          exit 1'
2026-03-14 09:44:27.270428 | controller |     - '        fi'
2026-03-14 09:44:27.270443 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.270450 | controller |     - '        '
2026-03-14 09:44:27.270456 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.270462 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.270469 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.270474 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.270480 | controller |       --timeout=300s'
2026-03-14 09:44:27.270486 | controller |     - '        '
2026-03-14 09:44:27.270492 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.270498 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.270504 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.270509 | controller |     - '        '
2026-03-14 09:44:27.270515 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.270521 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.270527 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.270533 | controller |     - '        '
2026-03-14 09:44:27.270538 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.270548 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.270554 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.270560 | controller |       available"'
2026-03-14 09:44:27.270565 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.270571 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.270577 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.270583 | controller |     - '          exit 1'
2026-03-14 09:44:27.270588 | controller |     - '        fi'
2026-03-14 09:44:27.270594 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.270600 | controller |     - '        '
2026-03-14 09:44:27.270606 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.270612 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.270618 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.270626 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.270632 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.270638 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.270644 | controller |       -- \'
2026-03-14 09:44:27.270650 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.270655 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.270661 | controller |     - '    logger.go:42: 09:28:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.270667 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.270673 | controller |     - '    logger.go:42: 09:28:08 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.270679 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.270684 | controller |       found'
2026-03-14 09:44:27.270690 | controller |     - '    logger.go:42: 09:28:09 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.270696 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.270702 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.270708 | controller |     - '        '
2026-03-14 09:44:27.270714 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.270720 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.270726 | controller |     - '        '
2026-03-14 09:44:27.270731 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.270737 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.270743 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.270749 | controller |     - '          exit 1'
2026-03-14 09:44:27.270755 | controller |     - '        fi'
2026-03-14 09:44:27.270761 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.270766 | controller |     - '        '
2026-03-14 09:44:27.270777 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.270783 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.270789 | controller |       | base64 -d)'
2026-03-14 09:44:27.270795 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.270801 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.270807 | controller |     - '          exit 1'
2026-03-14 09:44:27.270813 | controller |     - '        fi'
2026-03-14 09:44:27.270819 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.270825 | controller |     - '        '
2026-03-14 09:44:27.270842 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.270848 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.270854 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.270860 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.270866 | controller |       --timeout=300s'
2026-03-14 09:44:27.270872 | controller |     - '        '
2026-03-14 09:44:27.270879 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.270885 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.270891 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.270897 | controller |     - '        '
2026-03-14 09:44:27.270905 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.270911 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.270917 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.270923 | controller |     - '        '
2026-03-14 09:44:27.270939 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.270945 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.270951 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.270958 | controller |       available"'
2026-03-14 09:44:27.270964 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.270970 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.270976 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.270982 | controller |     - '          exit 1'
2026-03-14 09:44:27.270989 | controller |     - '        fi'
2026-03-14 09:44:27.270996 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.271002 | controller |     - '        '
2026-03-14 09:44:27.271009 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.271016 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.271022 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.271028 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.271034 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.271040 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.271046 | controller |       -- \'
2026-03-14 09:44:27.271052 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.271058 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.271063 | controller |     - '    logger.go:42: 09:28:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.271069 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.271075 | controller |     - '    logger.go:42: 09:28:10 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.271081 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.271087 | controller |       found'
2026-03-14 09:44:27.271093 | controller |     - '    logger.go:42: 09:28:11 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.271098 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.271104 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.271110 | controller |     - '        '
2026-03-14 09:44:27.271116 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.271121 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.271126 | controller |     - '        '
2026-03-14 09:44:27.271131 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.271136 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.271141 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.271146 | controller |     - '          exit 1'
2026-03-14 09:44:27.271151 | controller |     - '        fi'
2026-03-14 09:44:27.271156 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.271161 | controller |     - '        '
2026-03-14 09:44:27.271317 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.271327 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.271333 | controller |       | base64 -d)'
2026-03-14 09:44:27.271339 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.271345 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.271351 | controller |     - '          exit 1'
2026-03-14 09:44:27.271368 | controller |     - '        fi'
2026-03-14 09:44:27.271375 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.271381 | controller |     - '        '
2026-03-14 09:44:27.271387 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.271402 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.271408 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.271414 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.271420 | controller |       --timeout=300s'
2026-03-14 09:44:27.271426 | controller |     - '        '
2026-03-14 09:44:27.271432 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.271438 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.271443 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.271449 | controller |     - '        '
2026-03-14 09:44:27.271455 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.271461 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.271467 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.271473 | controller |     - '        '
2026-03-14 09:44:27.271479 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.271485 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.271491 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.271497 | controller |       available"'
2026-03-14 09:44:27.271502 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.271508 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.271514 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.271520 | controller |     - '          exit 1'
2026-03-14 09:44:27.271526 | controller |     - '        fi'
2026-03-14 09:44:27.271532 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.271537 | controller |     - '        '
2026-03-14 09:44:27.271543 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.271549 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.271555 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.271561 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.271566 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.271572 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.271578 | controller |       -- \'
2026-03-14 09:44:27.271584 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.271593 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.271599 | controller |     - '    logger.go:42: 09:28:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.271604 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.271610 | controller |     - '    logger.go:42: 09:28:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.271616 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.271622 | controller |       found'
2026-03-14 09:44:27.271628 | controller |     - '    logger.go:42: 09:28:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.271634 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.271640 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.271646 | controller |     - '        '
2026-03-14 09:44:27.271652 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.271658 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.271664 | controller |     - '        '
2026-03-14 09:44:27.271670 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.271675 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.271681 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.271704 | controller |     - '          exit 1'
2026-03-14 09:44:27.271714 | controller |     - '        fi'
2026-03-14 09:44:27.271720 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.271727 | controller |     - '        '
2026-03-14 09:44:27.271733 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.271739 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.271745 | controller |       | base64 -d)'
2026-03-14 09:44:27.271751 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.271757 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.271763 | controller |     - '          exit 1'
2026-03-14 09:44:27.271768 | controller |     - '        fi'
2026-03-14 09:44:27.271774 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.271780 | controller |     - '        '
2026-03-14 09:44:27.271786 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.271801 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.271808 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.271814 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.271820 | controller |       --timeout=300s'
2026-03-14 09:44:27.271826 | controller |     - '        '
2026-03-14 09:44:27.271832 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.271838 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.271844 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.271850 | controller |     - '        '
2026-03-14 09:44:27.271856 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.271861 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.271867 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.271873 | controller |     - '        '
2026-03-14 09:44:27.271879 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.271888 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.271907 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.271913 | controller |       available"'
2026-03-14 09:44:27.271921 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.271927 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.271931 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.271936 | controller |     - '          exit 1'
2026-03-14 09:44:27.271942 | controller |     - '        fi'
2026-03-14 09:44:27.271946 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.271951 | controller |     - '        '
2026-03-14 09:44:27.271955 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.271959 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.271963 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.271968 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.271972 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.271976 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.271980 | controller |       -- \'
2026-03-14 09:44:27.271985 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.271989 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.271993 | controller |     - '    logger.go:42: 09:28:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.271997 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.272002 | controller |     - '    logger.go:42: 09:28:12 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.272006 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.272010 | controller |       found'
2026-03-14 09:44:27.272015 | controller |     - '    logger.go:42: 09:28:13 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.272019 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.272023 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.272027 | controller |     - '        '
2026-03-14 09:44:27.272032 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.272036 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.272040 | controller |     - '        '
2026-03-14 09:44:27.272044 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.272049 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.272053 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.272057 | controller |     - '          exit 1'
2026-03-14 09:44:27.272061 | controller |     - '        fi'
2026-03-14 09:44:27.272066 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.272070 | controller |     - '        '
2026-03-14 09:44:27.272074 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.272078 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.272083 | controller |       | base64 -d)'
2026-03-14 09:44:27.272087 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.272091 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.272095 | controller |     - '          exit 1'
2026-03-14 09:44:27.272099 | controller |     - '        fi'
2026-03-14 09:44:27.272227 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.272234 | controller |     - '        '
2026-03-14 09:44:27.272239 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.272243 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.272254 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.272259 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.272264 | controller |       --timeout=300s'
2026-03-14 09:44:27.272269 | controller |     - '        '
2026-03-14 09:44:27.272273 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.272277 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.272284 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.272306 | controller |     - '        '
2026-03-14 09:44:27.272311 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.272315 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.272320 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.272324 | controller |     - '        '
2026-03-14 09:44:27.272328 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.272332 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.272336 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.272341 | controller |       available"'
2026-03-14 09:44:27.272345 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.272349 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.272353 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.272358 | controller |     - '          exit 1'
2026-03-14 09:44:27.272362 | controller |     - '        fi'
2026-03-14 09:44:27.272366 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.272370 | controller |     - '        '
2026-03-14 09:44:27.272374 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.272378 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.272383 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.272387 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.272391 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.272395 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.272400 | controller |       -- \'
2026-03-14 09:44:27.272404 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.272419 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.272425 | controller |     - '    logger.go:42: 09:28:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.272430 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.272434 | controller |     - '    logger.go:42: 09:28:13 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.272438 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.272453 | controller |       found'
2026-03-14 09:44:27.272457 | controller |     - '    logger.go:42: 09:28:14 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.272461 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.272465 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.272470 | controller |     - '        '
2026-03-14 09:44:27.272474 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.272479 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.272483 | controller |     - '        '
2026-03-14 09:44:27.272488 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.272492 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.272498 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.272502 | controller |     - '          exit 1'
2026-03-14 09:44:27.272507 | controller |     - '        fi'
2026-03-14 09:44:27.272511 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.272515 | controller |     - '        '
2026-03-14 09:44:27.272519 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.272524 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.272528 | controller |       | base64 -d)'
2026-03-14 09:44:27.272532 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.272536 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.272541 | controller |     - '          exit 1'
2026-03-14 09:44:27.272545 | controller |     - '        fi'
2026-03-14 09:44:27.272549 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.272553 | controller |     - '        '
2026-03-14 09:44:27.272558 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.272562 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.272577 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.272581 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.272586 | controller |       --timeout=300s'
2026-03-14 09:44:27.272590 | controller |     - '        '
2026-03-14 09:44:27.272594 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.272599 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.272603 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.272607 | controller |     - '        '
2026-03-14 09:44:27.272611 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.272616 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.272620 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.272624 | controller |     - '        '
2026-03-14 09:44:27.272628 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.272633 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.272637 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.272641 | controller |       available"'
2026-03-14 09:44:27.272645 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.272650 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.272657 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.272661 | controller |     - '          exit 1'
2026-03-14 09:44:27.272669 | controller |     - '        fi'
2026-03-14 09:44:27.272673 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.272679 | controller |     - '        '
2026-03-14 09:44:27.272683 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.272687 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.272692 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.272696 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.272700 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.272704 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.272709 | controller |       -- \'
2026-03-14 09:44:27.272713 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.272717 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.272722 | controller |     - '    logger.go:42: 09:28:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.272726 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.272730 | controller |     - '    logger.go:42: 09:28:14 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.272735 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.272739 | controller |       found'
2026-03-14 09:44:27.272743 | controller |     - '    logger.go:42: 09:28:15 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.272748 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.272752 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.272756 | controller |     - '        '
2026-03-14 09:44:27.272762 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.272768 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.272774 | controller |     - '        '
2026-03-14 09:44:27.272783 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.272789 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.272795 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.272801 | controller |     - '          exit 1'
2026-03-14 09:44:27.272806 | controller |     - '        fi'
2026-03-14 09:44:27.272812 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.272817 | controller |     - '        '
2026-03-14 09:44:27.272823 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.272828 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.272834 | controller |       | base64 -d)'
2026-03-14 09:44:27.272840 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.272846 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.272852 | controller |     - '          exit 1'
2026-03-14 09:44:27.272857 | controller |     - '        fi'
2026-03-14 09:44:27.272861 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.272865 | controller |     - '        '
2026-03-14 09:44:27.272870 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.272874 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.272878 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.272890 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.272894 | controller |       --timeout=300s'
2026-03-14 09:44:27.272901 | controller |     - '        '
2026-03-14 09:44:27.272906 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.272910 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.272915 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.272922 | controller |     - '        '
2026-03-14 09:44:27.272928 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.272936 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.272943 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.272949 | controller |     - '        '
2026-03-14 09:44:27.272955 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.272962 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.272968 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.272974 | controller |       available"'
2026-03-14 09:44:27.272979 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.272985 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.272990 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.272995 | controller |     - '          exit 1'
2026-03-14 09:44:27.272999 | controller |     - '        fi'
2026-03-14 09:44:27.273003 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.273007 | controller |     - '        '
2026-03-14 09:44:27.273012 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.273016 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.273020 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.273024 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.273028 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.273032 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.273037 | controller |       -- \'
2026-03-14 09:44:27.273041 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.273045 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.273049 | controller |     - '    logger.go:42: 09:28:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.273054 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.273058 | controller |     - '    logger.go:42: 09:28:15 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.273062 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.273066 | controller |       found'
2026-03-14 09:44:27.273070 | controller |     - '    logger.go:42: 09:28:16 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.273086 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.273090 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.273094 | controller |     - '        '
2026-03-14 09:44:27.273129 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.273134 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.273138 | controller |     - '        '
2026-03-14 09:44:27.273145 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.273149 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.273153 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.273157 | controller |     - '          exit 1'
2026-03-14 09:44:27.273161 | controller |     - '        fi'
2026-03-14 09:44:27.273166 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.273170 | controller |     - '        '
2026-03-14 09:44:27.273174 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.273178 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.273183 | controller |       | base64 -d)'
2026-03-14 09:44:27.273187 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.273191 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.273195 | controller |     - '          exit 1'
2026-03-14 09:44:27.273200 | controller |     - '        fi'
2026-03-14 09:44:27.273204 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.273208 | controller |     - '        '
2026-03-14 09:44:27.273212 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.273217 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.273221 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.273232 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.273237 | controller |       --timeout=300s'
2026-03-14 09:44:27.273241 | controller |     - '        '
2026-03-14 09:44:27.273246 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.273250 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.273254 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.273259 | controller |     - '        '
2026-03-14 09:44:27.273263 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.273267 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.273272 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.273276 | controller |     - '        '
2026-03-14 09:44:27.273280 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.273285 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.273306 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.273311 | controller |       available"'
2026-03-14 09:44:27.273315 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.273319 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.273324 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.273328 | controller |     - '          exit 1'
2026-03-14 09:44:27.273332 | controller |     - '        fi'
2026-03-14 09:44:27.273337 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.273341 | controller |     - '        '
2026-03-14 09:44:27.273349 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.273354 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.273358 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.273364 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.273370 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.273374 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.273379 | controller |       -- \'
2026-03-14 09:44:27.273383 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.273387 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.273391 | controller |     - '    logger.go:42: 09:28:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.273396 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.273400 | controller |     - '    logger.go:42: 09:28:16 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.273404 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.273409 | controller |       found'
2026-03-14 09:44:27.273413 | controller |     - '    logger.go:42: 09:28:17 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.273417 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.273422 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.273426 | controller |     - '        '
2026-03-14 09:44:27.273431 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.273435 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.273439 | controller |     - '        '
2026-03-14 09:44:27.273443 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.273448 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.273452 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.273456 | controller |     - '          exit 1'
2026-03-14 09:44:27.273461 | controller |     - '        fi'
2026-03-14 09:44:27.273465 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.273469 | controller |     - '        '
2026-03-14 09:44:27.273474 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.273478 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.273482 | controller |       | base64 -d)'
2026-03-14 09:44:27.273486 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.273491 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.273495 | controller |     - '          exit 1'
2026-03-14 09:44:27.273499 | controller |     - '        fi'
2026-03-14 09:44:27.273504 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.273508 | controller |     - '        '
2026-03-14 09:44:27.273512 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.273517 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.273521 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.273525 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.273529 | controller |       --timeout=300s'
2026-03-14 09:44:27.273534 | controller |     - '        '
2026-03-14 09:44:27.273545 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.273550 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.273554 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.273559 | controller |     - '        '
2026-03-14 09:44:27.273563 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.273569 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.273573 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.273578 | controller |     - '        '
2026-03-14 09:44:27.273582 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.273586 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.273591 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.273595 | controller |       available"'
2026-03-14 09:44:27.273599 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.273604 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.273608 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.273612 | controller |     - '          exit 1'
2026-03-14 09:44:27.273617 | controller |     - '        fi'
2026-03-14 09:44:27.273621 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.273625 | controller |     - '        '
2026-03-14 09:44:27.273629 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.273634 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.273638 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.273642 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.273647 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.273651 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.273655 | controller |       -- \'
2026-03-14 09:44:27.273660 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.273665 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.273669 | controller |     - '    logger.go:42: 09:28:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.273673 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.273678 | controller |     - '    logger.go:42: 09:28:17 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.273682 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.273686 | controller |       found'
2026-03-14 09:44:27.273690 | controller |     - '    logger.go:42: 09:28:18 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.273695 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.273699 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.273704 | controller |     - '        '
2026-03-14 09:44:27.273708 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.273712 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.273716 | controller |     - '        '
2026-03-14 09:44:27.273721 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.273725 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.273729 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.273735 | controller |     - '          exit 1'
2026-03-14 09:44:27.273741 | controller |     - '        fi'
2026-03-14 09:44:27.273747 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.273753 | controller |     - '        '
2026-03-14 09:44:27.273759 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.273768 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.273774 | controller |       | base64 -d)'
2026-03-14 09:44:27.273781 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.273786 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.273790 | controller |     - '          exit 1'
2026-03-14 09:44:27.273794 | controller |     - '        fi'
2026-03-14 09:44:27.273799 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.273803 | controller |     - '        '
2026-03-14 09:44:27.273807 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.273812 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.273818 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.273824 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.273830 | controller |       --timeout=300s'
2026-03-14 09:44:27.273836 | controller |     - '        '
2026-03-14 09:44:27.273871 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.273880 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.273887 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.273893 | controller |     - '        '
2026-03-14 09:44:27.273903 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.273910 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.273915 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.273921 | controller |     - '        '
2026-03-14 09:44:27.273926 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.273930 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.273935 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.273939 | controller |       available"'
2026-03-14 09:44:27.273945 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.273950 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.273954 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.273958 | controller |     - '          exit 1'
2026-03-14 09:44:27.273962 | controller |     - '        fi'
2026-03-14 09:44:27.273967 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.273971 | controller |     - '        '
2026-03-14 09:44:27.273975 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.273979 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.273983 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.273989 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.273994 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.273998 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.274002 | controller |       -- \'
2026-03-14 09:44:27.274006 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.274010 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.274017 | controller |     - '    logger.go:42: 09:28:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.274021 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.274025 | controller |     - '    logger.go:42: 09:28:19 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.274030 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.274034 | controller |       found'
2026-03-14 09:44:27.274038 | controller |     - '    logger.go:42: 09:28:20 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.274042 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.274046 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.274051 | controller |     - '        '
2026-03-14 09:44:27.274055 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.274059 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.274063 | controller |     - '        '
2026-03-14 09:44:27.274067 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.274072 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.274076 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.274080 | controller |     - '          exit 1'
2026-03-14 09:44:27.274084 | controller |     - '        fi'
2026-03-14 09:44:27.274089 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.274093 | controller |     - '        '
2026-03-14 09:44:27.274101 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.274105 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.274109 | controller |       | base64 -d)'
2026-03-14 09:44:27.274114 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.274118 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.274122 | controller |     - '          exit 1'
2026-03-14 09:44:27.274126 | controller |     - '        fi'
2026-03-14 09:44:27.274130 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.274135 | controller |     - '        '
2026-03-14 09:44:27.274139 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.274143 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.274147 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.274152 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.274156 | controller |       --timeout=300s'
2026-03-14 09:44:27.274160 | controller |     - '        '
2026-03-14 09:44:27.274164 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.274176 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.274181 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.274185 | controller |     - '        '
2026-03-14 09:44:27.274190 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.274194 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.274198 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.274202 | controller |     - '        '
2026-03-14 09:44:27.274207 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.274211 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.274217 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.274221 | controller |       available"'
2026-03-14 09:44:27.274226 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.274230 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.274234 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.274238 | controller |     - '          exit 1'
2026-03-14 09:44:27.274243 | controller |     - '        fi'
2026-03-14 09:44:27.274247 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.274251 | controller |     - '        '
2026-03-14 09:44:27.274255 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.274260 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.274264 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.274268 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.274272 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.274277 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.274281 | controller |       -- \'
2026-03-14 09:44:27.274285 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.274302 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.274309 | controller |     - '    logger.go:42: 09:28:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.274314 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.274318 | controller |     - '    logger.go:42: 09:28:20 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.274323 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.274327 | controller |       found'
2026-03-14 09:44:27.274331 | controller |     - '    logger.go:42: 09:28:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.274336 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.274340 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.274344 | controller |     - '        '
2026-03-14 09:44:27.274349 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.274353 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.274357 | controller |     - '        '
2026-03-14 09:44:27.274362 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.274366 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.274370 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.274375 | controller |     - '          exit 1'
2026-03-14 09:44:27.274379 | controller |     - '        fi'
2026-03-14 09:44:27.274383 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.274388 | controller |     - '        '
2026-03-14 09:44:27.274392 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.274396 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.274411 | controller |       | base64 -d)'
2026-03-14 09:44:27.274416 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.274420 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.274425 | controller |     - '          exit 1'
2026-03-14 09:44:27.274429 | controller |     - '        fi'
2026-03-14 09:44:27.274434 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.274439 | controller |     - '        '
2026-03-14 09:44:27.274444 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.274448 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.274452 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.274457 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.274461 | controller |       --timeout=300s'
2026-03-14 09:44:27.274465 | controller |     - '        '
2026-03-14 09:44:27.274470 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.274480 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.274485 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.274489 | controller |     - '        '
2026-03-14 09:44:27.274494 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.274500 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.274505 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.274509 | controller |     - '        '
2026-03-14 09:44:27.274513 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.274518 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.274522 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.274526 | controller |       available"'
2026-03-14 09:44:27.274531 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.274535 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.274539 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.274543 | controller |     - '          exit 1'
2026-03-14 09:44:27.274547 | controller |     - '        fi'
2026-03-14 09:44:27.274552 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.274556 | controller |     - '        '
2026-03-14 09:44:27.274560 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.274564 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.274568 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.274573 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.274577 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.274581 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.274585 | controller |       -- \'
2026-03-14 09:44:27.274590 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.274595 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.274600 | controller |     - '    logger.go:42: 09:28:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.274604 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.274608 | controller |     - '    logger.go:42: 09:28:21 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.274612 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.274616 | controller |       found'
2026-03-14 09:44:27.274621 | controller |     - '    logger.go:42: 09:28:22 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.274627 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.274631 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.274635 | controller |     - '        '
2026-03-14 09:44:27.274639 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.274644 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.274648 | controller |     - '        '
2026-03-14 09:44:27.274652 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.274656 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.274660 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.274665 | controller |     - '          exit 1'
2026-03-14 09:44:27.274669 | controller |     - '        fi'
2026-03-14 09:44:27.274673 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.274677 | controller |     - '        '
2026-03-14 09:44:27.274682 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.274686 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.274690 | controller |       | base64 -d)'
2026-03-14 09:44:27.274694 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.274698 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.274703 | controller |     - '          exit 1'
2026-03-14 09:44:27.274707 | controller |     - '        fi'
2026-03-14 09:44:27.274711 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.274715 | controller |     - '        '
2026-03-14 09:44:27.274719 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.274724 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.274728 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.274732 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.274736 | controller |       --timeout=300s'
2026-03-14 09:44:27.274741 | controller |     - '        '
2026-03-14 09:44:27.274745 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.274755 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.274760 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.274764 | controller |     - '        '
2026-03-14 09:44:27.274768 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.274772 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.274777 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.274781 | controller |     - '        '
2026-03-14 09:44:27.274785 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.274789 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.274793 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.274798 | controller |       available"'
2026-03-14 09:44:27.274802 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.274806 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.274810 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.274814 | controller |     - '          exit 1'
2026-03-14 09:44:27.274819 | controller |     - '        fi'
2026-03-14 09:44:27.274824 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.274829 | controller |     - '        '
2026-03-14 09:44:27.274833 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.274837 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.274841 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.274845 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.274850 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.274854 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.274858 | controller |       -- \'
2026-03-14 09:44:27.274862 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.274867 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.274871 | controller |     - '    logger.go:42: 09:28:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.274875 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.274879 | controller |     - '    logger.go:42: 09:28:22 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.274884 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.274888 | controller |       found'
2026-03-14 09:44:27.274895 | controller |     - '    logger.go:42: 09:28:23 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.274899 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.274903 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.274908 | controller |     - '        '
2026-03-14 09:44:27.274912 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.274916 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.274920 | controller |     - '        '
2026-03-14 09:44:27.274925 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.274929 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.274933 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.274937 | controller |     - '          exit 1'
2026-03-14 09:44:27.274942 | controller |     - '        fi'
2026-03-14 09:44:27.274946 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.274950 | controller |     - '        '
2026-03-14 09:44:27.274955 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.274959 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.274963 | controller |       | base64 -d)'
2026-03-14 09:44:27.274968 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.274972 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.274976 | controller |     - '          exit 1'
2026-03-14 09:44:27.274980 | controller |     - '        fi'
2026-03-14 09:44:27.274985 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.274989 | controller |     - '        '
2026-03-14 09:44:27.274995 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.274999 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.275004 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.275008 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.275012 | controller |       --timeout=300s'
2026-03-14 09:44:27.275017 | controller |     - '        '
2026-03-14 09:44:27.275023 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.275027 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.275037 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.275042 | controller |     - '        '
2026-03-14 09:44:27.275047 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.275051 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.275055 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.275060 | controller |     - '        '
2026-03-14 09:44:27.275064 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.275068 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.275073 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.275077 | controller |       available"'
2026-03-14 09:44:27.275081 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.275086 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.275090 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.275094 | controller |     - '          exit 1'
2026-03-14 09:44:27.275098 | controller |     - '        fi'
2026-03-14 09:44:27.275103 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.275107 | controller |     - '        '
2026-03-14 09:44:27.275111 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.275116 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.275120 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.275124 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.275128 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.275133 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.275137 | controller |       -- \'
2026-03-14 09:44:27.275141 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.275146 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.275150 | controller |     - '    logger.go:42: 09:28:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.275154 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.275168 | controller |     - '    logger.go:42: 09:28:23 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.275174 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.275178 | controller |       found'
2026-03-14 09:44:27.275183 | controller |     - '    logger.go:42: 09:28:24 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.275187 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.275191 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.275195 | controller |     - '        '
2026-03-14 09:44:27.275200 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.275204 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.275208 | controller |     - '        '
2026-03-14 09:44:27.275212 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.275219 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.275223 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.275227 | controller |     - '          exit 1'
2026-03-14 09:44:27.275232 | controller |     - '        fi'
2026-03-14 09:44:27.275236 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.275240 | controller |     - '        '
2026-03-14 09:44:27.275245 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.275249 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.275253 | controller |       | base64 -d)'
2026-03-14 09:44:27.275257 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.275264 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.275268 | controller |     - '          exit 1'
2026-03-14 09:44:27.275272 | controller |     - '        fi'
2026-03-14 09:44:27.275277 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.275281 | controller |     - '        '
2026-03-14 09:44:27.275285 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.275309 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.275316 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.275320 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.275325 | controller |       --timeout=300s'
2026-03-14 09:44:27.275329 | controller |     - '        '
2026-03-14 09:44:27.275333 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.275337 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.275348 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.275353 | controller |     - '        '
2026-03-14 09:44:27.275357 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.275361 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.275366 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.275370 | controller |     - '        '
2026-03-14 09:44:27.275374 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.275379 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.275383 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.275387 | controller |       available"'
2026-03-14 09:44:27.275391 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.275398 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.275402 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.275407 | controller |     - '          exit 1'
2026-03-14 09:44:27.275411 | controller |     - '        fi'
2026-03-14 09:44:27.275415 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.275420 | controller |     - '        '
2026-03-14 09:44:27.275424 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.275428 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.275433 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.275437 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.275451 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.275456 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.275463 | controller |       -- \'
2026-03-14 09:44:27.275468 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.275472 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.275476 | controller |     - '    logger.go:42: 09:28:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.275481 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.275485 | controller |     - '    logger.go:42: 09:28:24 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.275489 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.275494 | controller |       found'
2026-03-14 09:44:27.275498 | controller |     - '    logger.go:42: 09:28:25 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.275502 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.275507 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.275511 | controller |     - '        '
2026-03-14 09:44:27.275515 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.275520 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.275524 | controller |     - '        '
2026-03-14 09:44:27.275528 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.275533 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.275537 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.275541 | controller |     - '          exit 1'
2026-03-14 09:44:27.275546 | controller |     - '        fi'
2026-03-14 09:44:27.275550 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.275554 | controller |     - '        '
2026-03-14 09:44:27.275559 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.275563 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.275567 | controller |       | base64 -d)'
2026-03-14 09:44:27.275571 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.275576 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.275580 | controller |     - '          exit 1'
2026-03-14 09:44:27.275584 | controller |     - '        fi'
2026-03-14 09:44:27.275589 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.275593 | controller |     - '        '
2026-03-14 09:44:27.275597 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.275602 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.275606 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.275610 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.275615 | controller |       --timeout=300s'
2026-03-14 09:44:27.275622 | controller |     - '        '
2026-03-14 09:44:27.275626 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.275630 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.275635 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.275639 | controller |     - '        '
2026-03-14 09:44:27.275650 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.275656 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.275661 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.275665 | controller |     - '        '
2026-03-14 09:44:27.275670 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.275674 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.275678 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.275682 | controller |       available"'
2026-03-14 09:44:27.275687 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.275691 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.275695 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.275700 | controller |     - '          exit 1'
2026-03-14 09:44:27.275704 | controller |     - '        fi'
2026-03-14 09:44:27.275708 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.275713 | controller |     - '        '
2026-03-14 09:44:27.275717 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.275721 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.275726 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.275730 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.275734 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.275739 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.275743 | controller |       -- \'
2026-03-14 09:44:27.275747 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.275752 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.275756 | controller |     - '    logger.go:42: 09:28:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.275760 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.275765 | controller |     - '    logger.go:42: 09:28:25 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.275769 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.275773 | controller |       found'
2026-03-14 09:44:27.275778 | controller |     - '    logger.go:42: 09:28:26 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.275782 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.275788 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.275793 | controller |     - '        '
2026-03-14 09:44:27.275797 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.275801 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.275806 | controller |     - '        '
2026-03-14 09:44:27.275810 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.275814 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.275819 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.275823 | controller |     - '          exit 1'
2026-03-14 09:44:27.275827 | controller |     - '        fi'
2026-03-14 09:44:27.275832 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.275836 | controller |     - '        '
2026-03-14 09:44:27.275840 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.275845 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.275857 | controller |       | base64 -d)'
2026-03-14 09:44:27.275862 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.275866 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.275870 | controller |     - '          exit 1'
2026-03-14 09:44:27.275875 | controller |     - '        fi'
2026-03-14 09:44:27.275879 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.275883 | controller |     - '        '
2026-03-14 09:44:27.275888 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.275892 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.275896 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.275901 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.275905 | controller |       --timeout=300s'
2026-03-14 09:44:27.275909 | controller |     - '        '
2026-03-14 09:44:27.275914 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.275919 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.275925 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.275929 | controller |     - '        '
2026-03-14 09:44:27.275941 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.275946 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.275950 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.275955 | controller |     - '        '
2026-03-14 09:44:27.275959 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.275963 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.275968 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.275972 | controller |       available"'
2026-03-14 09:44:27.275976 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.275981 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.275985 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.275989 | controller |     - '          exit 1'
2026-03-14 09:44:27.275993 | controller |     - '        fi'
2026-03-14 09:44:27.275998 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.276002 | controller |     - '        '
2026-03-14 09:44:27.276006 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.276011 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.276015 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.276019 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.276024 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.276028 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.276032 | controller |       -- \'
2026-03-14 09:44:27.276036 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.276041 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.276047 | controller |     - '    logger.go:42: 09:28:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.276051 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.276055 | controller |     - '    logger.go:42: 09:28:27 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.276060 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.276064 | controller |       found'
2026-03-14 09:44:27.276068 | controller |     - '    logger.go:42: 09:28:28 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.276073 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.276077 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.276081 | controller |     - '        '
2026-03-14 09:44:27.276086 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.276090 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.276094 | controller |     - '        '
2026-03-14 09:44:27.276098 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.276103 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.276107 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.276111 | controller |     - '          exit 1'
2026-03-14 09:44:27.276116 | controller |     - '        fi'
2026-03-14 09:44:27.276123 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.276127 | controller |     - '        '
2026-03-14 09:44:27.276132 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.276136 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.276140 | controller |       | base64 -d)'
2026-03-14 09:44:27.276145 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.276149 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.276153 | controller |     - '          exit 1'
2026-03-14 09:44:27.276158 | controller |     - '        fi'
2026-03-14 09:44:27.276162 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.276166 | controller |     - '        '
2026-03-14 09:44:27.276171 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.276175 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.276179 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.276184 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.276188 | controller |       --timeout=300s'
2026-03-14 09:44:27.276192 | controller |     - '        '
2026-03-14 09:44:27.276197 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.276201 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.276205 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.276209 | controller |     - '        '
2026-03-14 09:44:27.276214 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.276224 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.276229 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.276233 | controller |     - '        '
2026-03-14 09:44:27.276237 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.276242 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.276246 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.276252 | controller |       available"'
2026-03-14 09:44:27.276257 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.276261 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.276265 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.276270 | controller |     - '          exit 1'
2026-03-14 09:44:27.276274 | controller |     - '        fi'
2026-03-14 09:44:27.276278 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.276283 | controller |     - '        '
2026-03-14 09:44:27.276290 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.276308 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.276312 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.276317 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.276321 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.276325 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.276329 | controller |       -- \'
2026-03-14 09:44:27.276334 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.276338 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.276342 | controller |     - '    logger.go:42: 09:28:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.276347 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.276351 | controller |     - '    logger.go:42: 09:28:28 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.276355 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.276360 | controller |       found'
2026-03-14 09:44:27.276374 | controller |     - '    logger.go:42: 09:28:29 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.276379 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.276391 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.276396 | controller |     - '        '
2026-03-14 09:44:27.276400 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.276405 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.276409 | controller |     - '        '
2026-03-14 09:44:27.276413 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.276418 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.276422 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.276426 | controller |     - '          exit 1'
2026-03-14 09:44:27.276431 | controller |     - '        fi'
2026-03-14 09:44:27.276435 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.276439 | controller |     - '        '
2026-03-14 09:44:27.276443 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.276448 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.276452 | controller |       | base64 -d)'
2026-03-14 09:44:27.276456 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.276460 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.276465 | controller |     - '          exit 1'
2026-03-14 09:44:27.276469 | controller |     - '        fi'
2026-03-14 09:44:27.276473 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.276478 | controller |     - '        '
2026-03-14 09:44:27.276484 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.276489 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.276493 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.276497 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.276501 | controller |       --timeout=300s'
2026-03-14 09:44:27.276506 | controller |     - '        '
2026-03-14 09:44:27.276510 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.276514 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.276518 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.276523 | controller |     - '        '
2026-03-14 09:44:27.276527 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.276537 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.276550 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.276555 | controller |     - '        '
2026-03-14 09:44:27.276560 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.276565 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.276569 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.276573 | controller |       available"'
2026-03-14 09:44:27.276578 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.276582 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.276586 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.276590 | controller |     - '          exit 1'
2026-03-14 09:44:27.276595 | controller |     - '        fi'
2026-03-14 09:44:27.276599 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.276603 | controller |     - '        '
2026-03-14 09:44:27.276607 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.276612 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.276616 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.276620 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.276624 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.276629 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.276633 | controller |       -- \'
2026-03-14 09:44:27.276637 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.276641 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.276648 | controller |     - '    logger.go:42: 09:28:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.276652 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.276656 | controller |     - '    logger.go:42: 09:28:29 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.276661 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.276665 | controller |       found'
2026-03-14 09:44:27.276669 | controller |     - '    logger.go:42: 09:28:30 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.276675 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.276680 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.276685 | controller |     - '        '
2026-03-14 09:44:27.276689 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.276693 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.276698 | controller |     - '        '
2026-03-14 09:44:27.276702 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.276706 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.276710 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.276715 | controller |     - '          exit 1'
2026-03-14 09:44:27.276719 | controller |     - '        fi'
2026-03-14 09:44:27.276723 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.276728 | controller |     - '        '
2026-03-14 09:44:27.276734 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.276738 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.276743 | controller |       | base64 -d)'
2026-03-14 09:44:27.276747 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.276751 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.276755 | controller |     - '          exit 1'
2026-03-14 09:44:27.276760 | controller |     - '        fi'
2026-03-14 09:44:27.276764 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.276768 | controller |     - '        '
2026-03-14 09:44:27.276772 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.276777 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.276781 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.276785 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.276789 | controller |       --timeout=300s'
2026-03-14 09:44:27.276794 | controller |     - '        '
2026-03-14 09:44:27.276799 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.276804 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.276810 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.276819 | controller |     - '        '
2026-03-14 09:44:27.276825 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.276831 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.276844 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.276850 | controller |     - '        '
2026-03-14 09:44:27.276855 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.276860 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.276864 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.276869 | controller |       available"'
2026-03-14 09:44:27.276873 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.276877 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.276881 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.276885 | controller |     - '          exit 1'
2026-03-14 09:44:27.276890 | controller |     - '        fi'
2026-03-14 09:44:27.276894 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.276910 | controller |     - '        '
2026-03-14 09:44:27.276914 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.276919 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.276923 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.276927 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.276931 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.276935 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.276940 | controller |       -- \'
2026-03-14 09:44:27.276944 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.276948 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.276952 | controller |     - '    logger.go:42: 09:28:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.276956 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.276961 | controller |     - '    logger.go:42: 09:28:30 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.276965 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.276969 | controller |       found'
2026-03-14 09:44:27.276973 | controller |     - '    logger.go:42: 09:28:31 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.276977 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.276982 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.276986 | controller |     - '        '
2026-03-14 09:44:27.276999 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.277004 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.277008 | controller |     - '        '
2026-03-14 09:44:27.277012 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.277016 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.277021 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.277025 | controller |     - '          exit 1'
2026-03-14 09:44:27.277029 | controller |     - '        fi'
2026-03-14 09:44:27.277033 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.277038 | controller |     - '        '
2026-03-14 09:44:27.277042 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.277046 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.277050 | controller |       | base64 -d)'
2026-03-14 09:44:27.277054 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.277058 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.277063 | controller |     - '          exit 1'
2026-03-14 09:44:27.277067 | controller |     - '        fi'
2026-03-14 09:44:27.277071 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.277075 | controller |     - '        '
2026-03-14 09:44:27.277079 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.277084 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.277088 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.277092 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.277096 | controller |       --timeout=300s'
2026-03-14 09:44:27.277100 | controller |     - '        '
2026-03-14 09:44:27.277105 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.277111 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.277115 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.277119 | controller |     - '        '
2026-03-14 09:44:27.277123 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.277127 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.277138 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.277142 | controller |     - '        '
2026-03-14 09:44:27.277147 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.277151 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.277155 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.277159 | controller |       available"'
2026-03-14 09:44:27.277164 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.277168 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.277172 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.277176 | controller |     - '          exit 1'
2026-03-14 09:44:27.277181 | controller |     - '        fi'
2026-03-14 09:44:27.277188 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.277192 | controller |     - '        '
2026-03-14 09:44:27.277196 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.277201 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.277205 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.277209 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.277213 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.277218 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.277222 | controller |       -- \'
2026-03-14 09:44:27.277226 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.277230 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.277235 | controller |     - '    logger.go:42: 09:28:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.277239 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.277243 | controller |     - '    logger.go:42: 09:28:31 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.277248 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.277252 | controller |       found'
2026-03-14 09:44:27.277256 | controller |     - '    logger.go:42: 09:28:32 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.277260 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.277265 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.277269 | controller |     - '        '
2026-03-14 09:44:27.277273 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.277277 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.277282 | controller |     - '        '
2026-03-14 09:44:27.277286 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.277306 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.277313 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.277317 | controller |     - '          exit 1'
2026-03-14 09:44:27.277322 | controller |     - '        fi'
2026-03-14 09:44:27.277326 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.277330 | controller |     - '        '
2026-03-14 09:44:27.277334 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.277339 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.277343 | controller |       | base64 -d)'
2026-03-14 09:44:27.277347 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.277352 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.277356 | controller |     - '          exit 1'
2026-03-14 09:44:27.277360 | controller |     - '        fi'
2026-03-14 09:44:27.277364 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.277369 | controller |     - '        '
2026-03-14 09:44:27.277373 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.277377 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.277382 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.277386 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.277390 | controller |       --timeout=300s'
2026-03-14 09:44:27.277394 | controller |     - '        '
2026-03-14 09:44:27.277399 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.277403 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.277407 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.277411 | controller |     - '        '
2026-03-14 09:44:27.277416 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.277420 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.277431 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.277436 | controller |     - '        '
2026-03-14 09:44:27.277440 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.277444 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.277448 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.277453 | controller |       available"'
2026-03-14 09:44:27.277457 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.277461 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.277466 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.277470 | controller |     - '          exit 1'
2026-03-14 09:44:27.277475 | controller |     - '        fi'
2026-03-14 09:44:27.277479 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.277483 | controller |     - '        '
2026-03-14 09:44:27.277488 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.277492 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.277496 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.277500 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.277505 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.277511 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.277515 | controller |       -- \'
2026-03-14 09:44:27.277520 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.277524 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.277530 | controller |     - '    logger.go:42: 09:28:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.277535 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.277539 | controller |     - '    logger.go:42: 09:28:32 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.277543 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.277547 | controller |       found'
2026-03-14 09:44:27.277552 | controller |     - '    logger.go:42: 09:28:33 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.277556 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.277560 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.277565 | controller |     - '        '
2026-03-14 09:44:27.277569 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.277573 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.277579 | controller |     - '        '
2026-03-14 09:44:27.277583 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.277588 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.277592 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.277596 | controller |     - '          exit 1'
2026-03-14 09:44:27.277600 | controller |     - '        fi'
2026-03-14 09:44:27.277605 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.277609 | controller |     - '        '
2026-03-14 09:44:27.277613 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.277618 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.277622 | controller |       | base64 -d)'
2026-03-14 09:44:27.277626 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.277630 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.277635 | controller |     - '          exit 1'
2026-03-14 09:44:27.277639 | controller |     - '        fi'
2026-03-14 09:44:27.277643 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.277648 | controller |     - '        '
2026-03-14 09:44:27.277652 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.277656 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.277661 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.277668 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.277674 | controller |       --timeout=300s'
2026-03-14 09:44:27.277680 | controller |     - '        '
2026-03-14 09:44:27.277686 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.277691 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.277696 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.277701 | controller |     - '        '
2026-03-14 09:44:27.277705 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.277709 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.277716 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.277720 | controller |     - '        '
2026-03-14 09:44:27.277732 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.277736 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.277741 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.277745 | controller |       available"'
2026-03-14 09:44:27.277749 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.277754 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.277758 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.277762 | controller |     - '          exit 1'
2026-03-14 09:44:27.277766 | controller |     - '        fi'
2026-03-14 09:44:27.277771 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.277775 | controller |     - '        '
2026-03-14 09:44:27.277779 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.277784 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.277788 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.277792 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.277796 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.277801 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.277805 | controller |       -- \'
2026-03-14 09:44:27.277809 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.277813 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.277818 | controller |     - '    logger.go:42: 09:28:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.277822 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.277826 | controller |     - '    logger.go:42: 09:28:33 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.277830 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.277834 | controller |       found'
2026-03-14 09:44:27.277839 | controller |     - '    logger.go:42: 09:28:34 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.277843 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.277848 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.277874 | controller |     - '        '
2026-03-14 09:44:27.277879 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.277883 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.277887 | controller |     - '        '
2026-03-14 09:44:27.277892 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.277896 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.277900 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.277904 | controller |     - '          exit 1'
2026-03-14 09:44:27.277909 | controller |     - '        fi'
2026-03-14 09:44:27.277913 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.277917 | controller |     - '        '
2026-03-14 09:44:27.277921 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.277925 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.277932 | controller |       | base64 -d)'
2026-03-14 09:44:27.277936 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.277940 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.277944 | controller |     - '          exit 1'
2026-03-14 09:44:27.277948 | controller |     - '        fi'
2026-03-14 09:44:27.277953 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.277957 | controller |     - '        '
2026-03-14 09:44:27.277961 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.277965 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.277970 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.277974 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.277978 | controller |       --timeout=300s'
2026-03-14 09:44:27.277982 | controller |     - '        '
2026-03-14 09:44:27.277987 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.277991 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.277995 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.277999 | controller |     - '        '
2026-03-14 09:44:27.278006 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.278010 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.278014 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.278018 | controller |     - '        '
2026-03-14 09:44:27.278023 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.278034 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.278038 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.278042 | controller |       available"'
2026-03-14 09:44:27.278047 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.278053 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.278057 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.278061 | controller |     - '          exit 1'
2026-03-14 09:44:27.278065 | controller |     - '        fi'
2026-03-14 09:44:27.278069 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.278074 | controller |     - '        '
2026-03-14 09:44:27.278078 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.278082 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.278086 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.278090 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.278095 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.278099 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.278103 | controller |       -- \'
2026-03-14 09:44:27.278107 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.278113 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.278118 | controller |     - '    logger.go:42: 09:28:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.278138 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.278149 | controller |     - '    logger.go:42: 09:28:34 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.278154 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.278158 | controller |       found'
2026-03-14 09:44:27.278162 | controller |     - '    logger.go:42: 09:28:35 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.278166 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.278170 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.278175 | controller |     - '        '
2026-03-14 09:44:27.278179 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.278183 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.278187 | controller |     - '        '
2026-03-14 09:44:27.278195 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.278199 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.278204 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.278208 | controller |     - '          exit 1'
2026-03-14 09:44:27.278212 | controller |     - '        fi'
2026-03-14 09:44:27.278216 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.278220 | controller |     - '        '
2026-03-14 09:44:27.278225 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.278229 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.278233 | controller |       | base64 -d)'
2026-03-14 09:44:27.278237 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.278241 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.278245 | controller |     - '          exit 1'
2026-03-14 09:44:27.278249 | controller |     - '        fi'
2026-03-14 09:44:27.278254 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.278258 | controller |     - '        '
2026-03-14 09:44:27.278262 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.278266 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.278270 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.278275 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.278279 | controller |       --timeout=300s'
2026-03-14 09:44:27.278283 | controller |     - '        '
2026-03-14 09:44:27.278308 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.278313 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.278317 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.278321 | controller |     - '        '
2026-03-14 09:44:27.278327 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.278333 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.278339 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.278345 | controller |     - '        '
2026-03-14 09:44:27.278350 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.278365 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.278372 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.278381 | controller |       available"'
2026-03-14 09:44:27.278387 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.278393 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.278397 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.278401 | controller |     - '          exit 1'
2026-03-14 09:44:27.278405 | controller |     - '        fi'
2026-03-14 09:44:27.278409 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.278414 | controller |     - '        '
2026-03-14 09:44:27.278418 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.278422 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.278426 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.278430 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.278434 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.278439 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.278443 | controller |       -- \'
2026-03-14 09:44:27.278447 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.278451 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.278455 | controller |     - '    logger.go:42: 09:28:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.278459 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.278464 | controller |     - '    logger.go:42: 09:28:36 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.278468 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.278472 | controller |       found'
2026-03-14 09:44:27.278476 | controller |     - '    logger.go:42: 09:28:37 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.278480 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.278484 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.278489 | controller |     - '        '
2026-03-14 09:44:27.278493 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.278497 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.278501 | controller |     - '        '
2026-03-14 09:44:27.278505 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.278509 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.278513 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.278517 | controller |     - '          exit 1'
2026-03-14 09:44:27.278522 | controller |     - '        fi'
2026-03-14 09:44:27.278526 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.278530 | controller |     - '        '
2026-03-14 09:44:27.278534 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.278538 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.278543 | controller |       | base64 -d)'
2026-03-14 09:44:27.278547 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.278551 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.278555 | controller |     - '          exit 1'
2026-03-14 09:44:27.278559 | controller |     - '        fi'
2026-03-14 09:44:27.278563 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.278567 | controller |     - '        '
2026-03-14 09:44:27.278572 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.278577 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.278582 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.278586 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.278590 | controller |       --timeout=300s'
2026-03-14 09:44:27.278594 | controller |     - '        '
2026-03-14 09:44:27.278598 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.278603 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.278607 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.278611 | controller |     - '        '
2026-03-14 09:44:27.278615 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.278619 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.278624 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.278628 | controller |     - '        '
2026-03-14 09:44:27.278632 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.278639 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.278650 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.278655 | controller |       available"'
2026-03-14 09:44:27.278659 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.278663 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.278668 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.278672 | controller |     - '          exit 1'
2026-03-14 09:44:27.278676 | controller |     - '        fi'
2026-03-14 09:44:27.278681 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.278685 | controller |     - '        '
2026-03-14 09:44:27.278689 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.278693 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.278698 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.278702 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.278706 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.278710 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.278715 | controller |       -- \'
2026-03-14 09:44:27.278719 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.278723 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.278727 | controller |     - '    logger.go:42: 09:28:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.278732 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.278736 | controller |     - '    logger.go:42: 09:28:37 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.278740 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.278744 | controller |       found'
2026-03-14 09:44:27.278749 | controller |     - '    logger.go:42: 09:28:38 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.278753 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.278759 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.278763 | controller |     - '        '
2026-03-14 09:44:27.278768 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.278772 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.278776 | controller |     - '        '
2026-03-14 09:44:27.278780 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.278785 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.278789 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.278793 | controller |     - '          exit 1'
2026-03-14 09:44:27.278798 | controller |     - '        fi'
2026-03-14 09:44:27.278802 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.278808 | controller |     - '        '
2026-03-14 09:44:27.278812 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.278816 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.278820 | controller |       | base64 -d)'
2026-03-14 09:44:27.278825 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.278829 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.278833 | controller |     - '          exit 1'
2026-03-14 09:44:27.278838 | controller |     - '        fi'
2026-03-14 09:44:27.278842 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.278846 | controller |     - '        '
2026-03-14 09:44:27.278850 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.278855 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.278859 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.278863 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.278867 | controller |       --timeout=300s'
2026-03-14 09:44:27.278872 | controller |     - '        '
2026-03-14 09:44:27.278876 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.278880 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.278884 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.278889 | controller |     - '        '
2026-03-14 09:44:27.278893 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.278897 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.278901 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.278906 | controller |     - '        '
2026-03-14 09:44:27.278910 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.278916 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.278927 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.278933 | controller |       available"'
2026-03-14 09:44:27.278938 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.278943 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.278947 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.278951 | controller |     - '          exit 1'
2026-03-14 09:44:27.278955 | controller |     - '        fi'
2026-03-14 09:44:27.278960 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.278964 | controller |     - '        '
2026-03-14 09:44:27.278970 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.278974 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.278978 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.278983 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.278987 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.278991 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.278996 | controller |       -- \'
2026-03-14 09:44:27.279000 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.279004 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.279008 | controller |     - '    logger.go:42: 09:28:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.279013 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.279017 | controller |     - '    logger.go:42: 09:28:38 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.279021 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.279025 | controller |       found'
2026-03-14 09:44:27.279030 | controller |     - '    logger.go:42: 09:28:39 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.279034 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.279038 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.279042 | controller |     - '        '
2026-03-14 09:44:27.279047 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.279051 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.279055 | controller |     - '        '
2026-03-14 09:44:27.279060 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.279064 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.279068 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.279072 | controller |     - '          exit 1'
2026-03-14 09:44:27.279077 | controller |     - '        fi'
2026-03-14 09:44:27.279081 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.279085 | controller |     - '        '
2026-03-14 09:44:27.279089 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.279094 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.279098 | controller |       | base64 -d)'
2026-03-14 09:44:27.279102 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.279106 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.279110 | controller |     - '          exit 1'
2026-03-14 09:44:27.279115 | controller |     - '        fi'
2026-03-14 09:44:27.279119 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.279123 | controller |     - '        '
2026-03-14 09:44:27.279127 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.279132 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.279136 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.279140 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.279144 | controller |       --timeout=300s'
2026-03-14 09:44:27.279149 | controller |     - '        '
2026-03-14 09:44:27.279153 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.279159 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.279163 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.279167 | controller |     - '        '
2026-03-14 09:44:27.279172 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.279176 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.279180 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.279184 | controller |     - '        '
2026-03-14 09:44:27.279189 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.279193 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.279197 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.279201 | controller |       available"'
2026-03-14 09:44:27.279211 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.279216 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.279220 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.279224 | controller |     - '          exit 1'
2026-03-14 09:44:27.279228 | controller |     - '        fi'
2026-03-14 09:44:27.279233 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.279237 | controller |     - '        '
2026-03-14 09:44:27.279243 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.279247 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.279252 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.279256 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.279260 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.279264 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.279269 | controller |       -- \'
2026-03-14 09:44:27.279273 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.279277 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.279282 | controller |     - '    logger.go:42: 09:28:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.279286 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.279306 | controller |     - '    logger.go:42: 09:28:39 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.279311 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.279316 | controller |       found'
2026-03-14 09:44:27.279320 | controller |     - '    logger.go:42: 09:28:40 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.279324 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.279329 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.279333 | controller |     - '        '
2026-03-14 09:44:27.279337 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.279341 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.279346 | controller |     - '        '
2026-03-14 09:44:27.279350 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.279354 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.279359 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.279365 | controller |     - '          exit 1'
2026-03-14 09:44:27.279369 | controller |     - '        fi'
2026-03-14 09:44:27.279373 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.279378 | controller |     - '        '
2026-03-14 09:44:27.279384 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.279388 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.279392 | controller |       | base64 -d)'
2026-03-14 09:44:27.279398 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.279402 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.279408 | controller |     - '          exit 1'
2026-03-14 09:44:27.279412 | controller |     - '        fi'
2026-03-14 09:44:27.279416 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.279421 | controller |     - '        '
2026-03-14 09:44:27.279425 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.279429 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.279435 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.279439 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.279443 | controller |       --timeout=300s'
2026-03-14 09:44:27.279447 | controller |     - '        '
2026-03-14 09:44:27.279452 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.279456 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.279460 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.279465 | controller |     - '        '
2026-03-14 09:44:27.279469 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.279473 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.279477 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.279482 | controller |     - '        '
2026-03-14 09:44:27.279486 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.279490 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.279494 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.279499 | controller |       available"'
2026-03-14 09:44:27.279503 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.279513 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.279518 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.279523 | controller |     - '          exit 1'
2026-03-14 09:44:27.279528 | controller |     - '        fi'
2026-03-14 09:44:27.279532 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.279536 | controller |     - '        '
2026-03-14 09:44:27.279541 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.279545 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.279549 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.279554 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.279558 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.279564 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.279568 | controller |       -- \'
2026-03-14 09:44:27.279572 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.279577 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.279581 | controller |     - '    logger.go:42: 09:28:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.279585 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.279590 | controller |     - '    logger.go:42: 09:28:40 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.279594 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.279598 | controller |       found'
2026-03-14 09:44:27.279603 | controller |     - '    logger.go:42: 09:28:41 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.279607 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.279611 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.279616 | controller |     - '        '
2026-03-14 09:44:27.279620 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.279624 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.279628 | controller |     - '        '
2026-03-14 09:44:27.279633 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.279637 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.279641 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.279646 | controller |     - '          exit 1'
2026-03-14 09:44:27.279650 | controller |     - '        fi'
2026-03-14 09:44:27.279654 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.279658 | controller |     - '        '
2026-03-14 09:44:27.279663 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.279667 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.279672 | controller |       | base64 -d)'
2026-03-14 09:44:27.279676 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.279680 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.279684 | controller |     - '          exit 1'
2026-03-14 09:44:27.279689 | controller |     - '        fi'
2026-03-14 09:44:27.279693 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.279697 | controller |     - '        '
2026-03-14 09:44:27.279703 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.279708 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.279724 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.279732 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.279738 | controller |       --timeout=300s'
2026-03-14 09:44:27.279750 | controller |     - '        '
2026-03-14 09:44:27.279756 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.279763 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.279769 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.279774 | controller |     - '        '
2026-03-14 09:44:27.279779 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.279783 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.279790 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.279794 | controller |     - '        '
2026-03-14 09:44:27.279799 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.279803 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.279807 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.279811 | controller |       available"'
2026-03-14 09:44:27.279816 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.279829 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.279835 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.279839 | controller |     - '          exit 1'
2026-03-14 09:44:27.279846 | controller |     - '        fi'
2026-03-14 09:44:27.279850 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.279855 | controller |     - '        '
2026-03-14 09:44:27.279859 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.279863 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.279867 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.279872 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.279876 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.279880 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.279884 | controller |       -- \'
2026-03-14 09:44:27.279889 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.279893 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.279897 | controller |     - '    logger.go:42: 09:28:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.279901 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.279906 | controller |     - '    logger.go:42: 09:28:41 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.279910 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.279914 | controller |       found'
2026-03-14 09:44:27.279918 | controller |     - '    logger.go:42: 09:28:42 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.279923 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.279927 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.279931 | controller |     - '        '
2026-03-14 09:44:27.279935 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.279940 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.279944 | controller |     - '        '
2026-03-14 09:44:27.279948 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.279953 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.279957 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.279961 | controller |     - '          exit 1'
2026-03-14 09:44:27.279966 | controller |     - '        fi'
2026-03-14 09:44:27.279970 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.279976 | controller |     - '        '
2026-03-14 09:44:27.279981 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.279985 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.279989 | controller |       | base64 -d)'
2026-03-14 09:44:27.279995 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.280000 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.280004 | controller |     - '          exit 1'
2026-03-14 09:44:27.280008 | controller |     - '        fi'
2026-03-14 09:44:27.280012 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.280017 | controller |     - '        '
2026-03-14 09:44:27.280021 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.280025 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.280030 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.280034 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.280040 | controller |       --timeout=300s'
2026-03-14 09:44:27.280044 | controller |     - '        '
2026-03-14 09:44:27.280049 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.280053 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.280057 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.280061 | controller |     - '        '
2026-03-14 09:44:27.280066 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.280070 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.280074 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.280078 | controller |     - '        '
2026-03-14 09:44:27.280083 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.280087 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.280091 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.280096 | controller |       available"'
2026-03-14 09:44:27.280100 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.280104 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.280114 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.280119 | controller |     - '          exit 1'
2026-03-14 09:44:27.280123 | controller |     - '        fi'
2026-03-14 09:44:27.280128 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.280132 | controller |     - '        '
2026-03-14 09:44:27.280136 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.280140 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.280145 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.280149 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.280153 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.280158 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.280162 | controller |       -- \'
2026-03-14 09:44:27.280166 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.280170 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.280175 | controller |     - '    logger.go:42: 09:28:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.280190 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.280195 | controller |     - '    logger.go:42: 09:28:42 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.280200 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.280205 | controller |       found'
2026-03-14 09:44:27.280209 | controller |     - '    logger.go:42: 09:28:43 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.280213 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.280218 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.280222 | controller |     - '        '
2026-03-14 09:44:27.280226 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.280230 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.280235 | controller |     - '        '
2026-03-14 09:44:27.280239 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.280243 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.280247 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.280252 | controller |     - '          exit 1'
2026-03-14 09:44:27.280256 | controller |     - '        fi'
2026-03-14 09:44:27.280260 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.280265 | controller |     - '        '
2026-03-14 09:44:27.280269 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.280273 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.280277 | controller |       | base64 -d)'
2026-03-14 09:44:27.280281 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.280286 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.280308 | controller |     - '          exit 1'
2026-03-14 09:44:27.280314 | controller |     - '        fi'
2026-03-14 09:44:27.280318 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.280323 | controller |     - '        '
2026-03-14 09:44:27.280327 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.280331 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.280335 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.280340 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.280344 | controller |       --timeout=300s'
2026-03-14 09:44:27.280348 | controller |     - '        '
2026-03-14 09:44:27.280352 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.280357 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.280361 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.280365 | controller |     - '        '
2026-03-14 09:44:27.280369 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.280374 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.280378 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.280382 | controller |     - '        '
2026-03-14 09:44:27.280386 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.280391 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.280395 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.280399 | controller |       available"'
2026-03-14 09:44:27.280403 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.280410 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.280414 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.280424 | controller |     - '          exit 1'
2026-03-14 09:44:27.280428 | controller |     - '        fi'
2026-03-14 09:44:27.280433 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.280437 | controller |     - '        '
2026-03-14 09:44:27.280441 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.280446 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.280450 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.280454 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.280458 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.280463 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.280467 | controller |       -- \'
2026-03-14 09:44:27.280471 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.280475 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.280480 | controller |     - '    logger.go:42: 09:28:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.280484 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.280488 | controller |     - '    logger.go:42: 09:28:44 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.280493 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.280497 | controller |       found'
2026-03-14 09:44:27.280502 | controller |     - '    logger.go:42: 09:28:45 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.280506 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.280510 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.280514 | controller |     - '        '
2026-03-14 09:44:27.280521 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.280525 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.280529 | controller |     - '        '
2026-03-14 09:44:27.280534 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.280538 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.280542 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.280547 | controller |     - '          exit 1'
2026-03-14 09:44:27.280551 | controller |     - '        fi'
2026-03-14 09:44:27.280555 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.280559 | controller |     - '        '
2026-03-14 09:44:27.280564 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.280568 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.280572 | controller |       | base64 -d)'
2026-03-14 09:44:27.280576 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.280581 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.280585 | controller |     - '          exit 1'
2026-03-14 09:44:27.280589 | controller |     - '        fi'
2026-03-14 09:44:27.280593 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.280598 | controller |     - '        '
2026-03-14 09:44:27.280602 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.280607 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.280619 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.280635 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.280641 | controller |       --timeout=300s'
2026-03-14 09:44:27.280646 | controller |     - '        '
2026-03-14 09:44:27.280650 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.280655 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.280659 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.280663 | controller |     - '        '
2026-03-14 09:44:27.280668 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.280672 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.280676 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.280680 | controller |     - '        '
2026-03-14 09:44:27.280689 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.280693 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.280698 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.280702 | controller |       available"'
2026-03-14 09:44:27.280706 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.280712 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.280716 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.280721 | controller |     - '          exit 1'
2026-03-14 09:44:27.280725 | controller |     - '        fi'
2026-03-14 09:44:27.280738 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.280743 | controller |     - '        '
2026-03-14 09:44:27.280748 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.280752 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.280756 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.280761 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.280767 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.280771 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.280775 | controller |       -- \'
2026-03-14 09:44:27.280779 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.280784 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.280789 | controller |     - '    logger.go:42: 09:28:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.280794 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.280798 | controller |     - '    logger.go:42: 09:28:45 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.280802 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.280806 | controller |       found'
2026-03-14 09:44:27.280811 | controller |     - '    logger.go:42: 09:28:46 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.280815 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.280819 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.280826 | controller |     - '        '
2026-03-14 09:44:27.280830 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.280835 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.280839 | controller |     - '        '
2026-03-14 09:44:27.280843 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.280847 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.280852 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.280856 | controller |     - '          exit 1'
2026-03-14 09:44:27.280860 | controller |     - '        fi'
2026-03-14 09:44:27.280864 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.280868 | controller |     - '        '
2026-03-14 09:44:27.280873 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.280877 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.280881 | controller |       | base64 -d)'
2026-03-14 09:44:27.280885 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.280889 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.280893 | controller |     - '          exit 1'
2026-03-14 09:44:27.280898 | controller |     - '        fi'
2026-03-14 09:44:27.280902 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.280906 | controller |     - '        '
2026-03-14 09:44:27.280910 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.280914 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.280918 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.280923 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.280927 | controller |       --timeout=300s'
2026-03-14 09:44:27.280931 | controller |     - '        '
2026-03-14 09:44:27.280935 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.280939 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.280943 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.280948 | controller |     - '        '
2026-03-14 09:44:27.280952 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.280956 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.280960 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.280964 | controller |     - '        '
2026-03-14 09:44:27.280968 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.280973 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.280977 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.280981 | controller |       available"'
2026-03-14 09:44:27.280985 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.280989 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.280993 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.280997 | controller |     - '          exit 1'
2026-03-14 09:44:27.281002 | controller |     - '        fi'
2026-03-14 09:44:27.281012 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.281016 | controller |     - '        '
2026-03-14 09:44:27.281021 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.281031 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.281037 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.281043 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.281049 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.281054 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.281060 | controller |       -- \'
2026-03-14 09:44:27.281065 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.281071 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.281076 | controller |     - '    logger.go:42: 09:28:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.281082 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.281088 | controller |     - '    logger.go:42: 09:28:46 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.281093 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.281099 | controller |       found'
2026-03-14 09:44:27.281104 | controller |     - '    logger.go:42: 09:28:47 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.281109 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.281114 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.281120 | controller |     - '        '
2026-03-14 09:44:27.281125 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.281130 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.281136 | controller |     - '        '
2026-03-14 09:44:27.281141 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.281147 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.281152 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.281158 | controller |     - '          exit 1'
2026-03-14 09:44:27.281163 | controller |     - '        fi'
2026-03-14 09:44:27.281169 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.281175 | controller |     - '        '
2026-03-14 09:44:27.281180 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.281186 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.281192 | controller |       | base64 -d)'
2026-03-14 09:44:27.281197 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.281203 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.281208 | controller |     - '          exit 1'
2026-03-14 09:44:27.281214 | controller |     - '        fi'
2026-03-14 09:44:27.281220 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.281225 | controller |     - '        '
2026-03-14 09:44:27.281231 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.281237 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.281242 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.281248 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.281254 | controller |       --timeout=300s'
2026-03-14 09:44:27.281260 | controller |     - '        '
2026-03-14 09:44:27.281266 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.281272 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.281282 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.281304 | controller |     - '        '
2026-03-14 09:44:27.281314 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.281320 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.281326 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.281332 | controller |     - '        '
2026-03-14 09:44:27.281338 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.281344 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.281353 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.281359 | controller |       available"'
2026-03-14 09:44:27.281365 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.281370 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.281376 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.281382 | controller |     - '          exit 1'
2026-03-14 09:44:27.281388 | controller |     - '        fi'
2026-03-14 09:44:27.281394 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.281400 | controller |     - '        '
2026-03-14 09:44:27.281415 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.281421 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.281427 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.281433 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.281439 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.281445 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.281451 | controller |       -- \'
2026-03-14 09:44:27.281456 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.281462 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.281468 | controller |     - '    logger.go:42: 09:28:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.281474 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.281479 | controller |     - '    logger.go:42: 09:28:47 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.281485 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.281491 | controller |       found'
2026-03-14 09:44:27.281497 | controller |     - '    logger.go:42: 09:28:48 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.281503 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.281509 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.281515 | controller |     - '        '
2026-03-14 09:44:27.281521 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.281527 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.281533 | controller |     - '        '
2026-03-14 09:44:27.281539 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.281545 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.281550 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.281559 | controller |     - '          exit 1'
2026-03-14 09:44:27.281565 | controller |     - '        fi'
2026-03-14 09:44:27.281571 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.281577 | controller |     - '        '
2026-03-14 09:44:27.281583 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.281588 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.281594 | controller |       | base64 -d)'
2026-03-14 09:44:27.281600 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.281605 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.281611 | controller |     - '          exit 1'
2026-03-14 09:44:27.281617 | controller |     - '        fi'
2026-03-14 09:44:27.281623 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.281628 | controller |     - '        '
2026-03-14 09:44:27.281634 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.281640 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.281646 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.281653 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.281659 | controller |       --timeout=300s'
2026-03-14 09:44:27.281665 | controller |     - '        '
2026-03-14 09:44:27.281671 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.281677 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.281683 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.281689 | controller |     - '        '
2026-03-14 09:44:27.281695 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.281701 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.281707 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.281713 | controller |     - '        '
2026-03-14 09:44:27.281722 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.281728 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.281735 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.281741 | controller |       available"'
2026-03-14 09:44:27.281747 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.281753 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.281759 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.281765 | controller |     - '          exit 1'
2026-03-14 09:44:27.281771 | controller |     - '        fi'
2026-03-14 09:44:27.281777 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.281783 | controller |     - '        '
2026-03-14 09:44:27.281789 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.281806 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.281813 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.281819 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.281825 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.281831 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.281840 | controller |       -- \'
2026-03-14 09:44:27.281846 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.281852 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.281858 | controller |     - '    logger.go:42: 09:28:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.281941 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.281948 | controller |     - '    logger.go:42: 09:28:48 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.281954 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.281960 | controller |       found'
2026-03-14 09:44:27.281966 | controller |     - '    logger.go:42: 09:28:49 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.281972 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.281983 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.281989 | controller |     - '        '
2026-03-14 09:44:27.281995 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.282001 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.282007 | controller |     - '        '
2026-03-14 09:44:27.282013 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.282019 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.282025 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.282031 | controller |     - '          exit 1'
2026-03-14 09:44:27.282037 | controller |     - '        fi'
2026-03-14 09:44:27.282043 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.282049 | controller |     - '        '
2026-03-14 09:44:27.282055 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.282061 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.282067 | controller |       | base64 -d)'
2026-03-14 09:44:27.282073 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.282079 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.282084 | controller |     - '          exit 1'
2026-03-14 09:44:27.282090 | controller |     - '        fi'
2026-03-14 09:44:27.282096 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.282102 | controller |     - '        '
2026-03-14 09:44:27.282108 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.282114 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.282120 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.282126 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.282132 | controller |       --timeout=300s'
2026-03-14 09:44:27.282138 | controller |     - '        '
2026-03-14 09:44:27.282144 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.282150 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.282156 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.282162 | controller |     - '        '
2026-03-14 09:44:27.282168 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.282186 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.282193 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.282201 | controller |     - '        '
2026-03-14 09:44:27.282207 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.282213 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.282219 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.282225 | controller |       available"'
2026-03-14 09:44:27.282231 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.282239 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.282245 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.282251 | controller |     - '          exit 1'
2026-03-14 09:44:27.282257 | controller |     - '        fi'
2026-03-14 09:44:27.282263 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.282269 | controller |     - '        '
2026-03-14 09:44:27.282275 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.282306 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.282317 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.282323 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.282329 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.282335 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.282341 | controller |       -- \'
2026-03-14 09:44:27.282347 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.282353 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.282359 | controller |     - '    logger.go:42: 09:28:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.282365 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.282371 | controller |     - '    logger.go:42: 09:28:49 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.282377 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.282383 | controller |       found'
2026-03-14 09:44:27.282389 | controller |     - '    logger.go:42: 09:28:50 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.282395 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.282401 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.282407 | controller |     - '        '
2026-03-14 09:44:27.282413 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.282419 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.282425 | controller |     - '        '
2026-03-14 09:44:27.282431 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.282436 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.282443 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.282449 | controller |     - '          exit 1'
2026-03-14 09:44:27.282455 | controller |     - '        fi'
2026-03-14 09:44:27.282461 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.282467 | controller |     - '        '
2026-03-14 09:44:27.282475 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.282482 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.282488 | controller |       | base64 -d)'
2026-03-14 09:44:27.282494 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.282510 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.282517 | controller |     - '          exit 1'
2026-03-14 09:44:27.282523 | controller |     - '        fi'
2026-03-14 09:44:27.282529 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.282535 | controller |     - '        '
2026-03-14 09:44:27.282541 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.282547 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.282553 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.282559 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.282565 | controller |       --timeout=300s'
2026-03-14 09:44:27.282570 | controller |     - '        '
2026-03-14 09:44:27.282576 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.282582 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.282587 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.282593 | controller |     - '        '
2026-03-14 09:44:27.282601 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.282607 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.282613 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.282619 | controller |     - '        '
2026-03-14 09:44:27.282625 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.282631 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.282637 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.282643 | controller |       available"'
2026-03-14 09:44:27.282649 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.282655 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.282661 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.282667 | controller |     - '          exit 1'
2026-03-14 09:44:27.282673 | controller |     - '        fi'
2026-03-14 09:44:27.282680 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.282686 | controller |     - '        '
2026-03-14 09:44:27.282692 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.282697 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.282711 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.282717 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.282723 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.282729 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.282735 | controller |       -- \'
2026-03-14 09:44:27.282741 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.282747 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.282753 | controller |     - '    logger.go:42: 09:28:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.282759 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.282768 | controller |     - '    logger.go:42: 09:28:50 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.282774 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.282780 | controller |       found'
2026-03-14 09:44:27.282786 | controller |     - '    logger.go:42: 09:28:51 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.282792 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.282798 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.282804 | controller |     - '        '
2026-03-14 09:44:27.282810 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.282816 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.282822 | controller |     - '        '
2026-03-14 09:44:27.282828 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.282834 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.282840 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.282846 | controller |     - '          exit 1'
2026-03-14 09:44:27.282852 | controller |     - '        fi'
2026-03-14 09:44:27.282858 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.282864 | controller |     - '        '
2026-03-14 09:44:27.282870 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.282877 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.282883 | controller |       | base64 -d)'
2026-03-14 09:44:27.282889 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.282895 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.282901 | controller |     - '          exit 1'
2026-03-14 09:44:27.282907 | controller |     - '        fi'
2026-03-14 09:44:27.282913 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.282920 | controller |     - '        '
2026-03-14 09:44:27.282927 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.282933 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.282941 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.282948 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.282954 | controller |       --timeout=300s'
2026-03-14 09:44:27.282960 | controller |     - '        '
2026-03-14 09:44:27.282966 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.282972 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.282978 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.282984 | controller |     - '        '
2026-03-14 09:44:27.282990 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.282996 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.283002 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.283007 | controller |     - '        '
2026-03-14 09:44:27.283013 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.283019 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.283025 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.283031 | controller |       available"'
2026-03-14 09:44:27.283037 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.283046 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.283052 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.283058 | controller |     - '          exit 1'
2026-03-14 09:44:27.283064 | controller |     - '        fi'
2026-03-14 09:44:27.283070 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.283076 | controller |     - '        '
2026-03-14 09:44:27.283099 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.283105 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.283118 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.283124 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.283130 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.283136 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.283142 | controller |       -- \'
2026-03-14 09:44:27.283148 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.283154 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.283160 | controller |     - '    logger.go:42: 09:28:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.283166 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.283173 | controller |     - '    logger.go:42: 09:28:51 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.283179 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.283185 | controller |       found'
2026-03-14 09:44:27.283190 | controller |     - '    logger.go:42: 09:28:52 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.283196 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.283202 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.283208 | controller |     - '        '
2026-03-14 09:44:27.283214 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.283220 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.283226 | controller |     - '        '
2026-03-14 09:44:27.283233 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.283239 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.283245 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.283251 | controller |     - '          exit 1'
2026-03-14 09:44:27.283257 | controller |     - '        fi'
2026-03-14 09:44:27.283263 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.283269 | controller |     - '        '
2026-03-14 09:44:27.283275 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.283282 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.283303 | controller |       | base64 -d)'
2026-03-14 09:44:27.283312 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.283318 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.283324 | controller |     - '          exit 1'
2026-03-14 09:44:27.283330 | controller |     - '        fi'
2026-03-14 09:44:27.283339 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.283346 | controller |     - '        '
2026-03-14 09:44:27.283352 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.283358 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.283364 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.283381 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.283387 | controller |       --timeout=300s'
2026-03-14 09:44:27.283393 | controller |     - '        '
2026-03-14 09:44:27.283399 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.283405 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.283411 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.283417 | controller |     - '        '
2026-03-14 09:44:27.283423 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.283429 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.283435 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.283441 | controller |     - '        '
2026-03-14 09:44:27.283447 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.283453 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.283459 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.283464 | controller |       available"'
2026-03-14 09:44:27.283470 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.283476 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.283482 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.283488 | controller |     - '          exit 1'
2026-03-14 09:44:27.283494 | controller |     - '        fi'
2026-03-14 09:44:27.283500 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.283506 | controller |     - '        '
2026-03-14 09:44:27.283512 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.283518 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.283524 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.283538 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.283544 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.283550 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.283567 | controller |       -- \'
2026-03-14 09:44:27.283574 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.283580 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.283585 | controller |     - '    logger.go:42: 09:28:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.283591 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.283597 | controller |     - '    logger.go:42: 09:28:53 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.283602 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.283608 | controller |       found'
2026-03-14 09:44:27.283614 | controller |     - '    logger.go:42: 09:28:54 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.283620 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.283625 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.283632 | controller |     - '        '
2026-03-14 09:44:27.283637 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.283646 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.283651 | controller |     - '        '
2026-03-14 09:44:27.283657 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.283662 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.283668 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.283673 | controller |     - '          exit 1'
2026-03-14 09:44:27.283679 | controller |     - '        fi'
2026-03-14 09:44:27.283684 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.283690 | controller |     - '        '
2026-03-14 09:44:27.283695 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.283701 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.283707 | controller |       | base64 -d)'
2026-03-14 09:44:27.283713 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.283719 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.283725 | controller |     - '          exit 1'
2026-03-14 09:44:27.283730 | controller |     - '        fi'
2026-03-14 09:44:27.283736 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.283741 | controller |     - '        '
2026-03-14 09:44:27.283745 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.283749 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.283755 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.283761 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.283767 | controller |       --timeout=300s'
2026-03-14 09:44:27.283773 | controller |     - '        '
2026-03-14 09:44:27.283779 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.283784 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.283789 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.283793 | controller |     - '        '
2026-03-14 09:44:27.283797 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.283801 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.283806 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.283810 | controller |     - '        '
2026-03-14 09:44:27.283814 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.283818 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.283822 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.283827 | controller |       available"'
2026-03-14 09:44:27.283831 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.283835 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.283839 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.283844 | controller |     - '          exit 1'
2026-03-14 09:44:27.283848 | controller |     - '        fi'
2026-03-14 09:44:27.283852 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.283856 | controller |     - '        '
2026-03-14 09:44:27.283860 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.283865 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.283881 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.283897 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.283902 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.283906 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.283910 | controller |       -- \'
2026-03-14 09:44:27.283914 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.283918 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.283923 | controller |     - '    logger.go:42: 09:28:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.283927 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.283931 | controller |     - '    logger.go:42: 09:28:54 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.283935 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.283940 | controller |       found'
2026-03-14 09:44:27.283944 | controller |     - '    logger.go:42: 09:28:55 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.283948 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.283952 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.283956 | controller |     - '        '
2026-03-14 09:44:27.283961 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.283965 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.283969 | controller |     - '        '
2026-03-14 09:44:27.283973 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.283977 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.283982 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.283986 | controller |     - '          exit 1'
2026-03-14 09:44:27.283990 | controller |     - '        fi'
2026-03-14 09:44:27.283994 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.283998 | controller |     - '        '
2026-03-14 09:44:27.284003 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.284007 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.284011 | controller |       | base64 -d)'
2026-03-14 09:44:27.284015 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.284019 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.284024 | controller |     - '          exit 1'
2026-03-14 09:44:27.284028 | controller |     - '        fi'
2026-03-14 09:44:27.284032 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.284036 | controller |     - '        '
2026-03-14 09:44:27.284041 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.284045 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.284049 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.284053 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.284057 | controller |       --timeout=300s'
2026-03-14 09:44:27.284061 | controller |     - '        '
2026-03-14 09:44:27.284066 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.284070 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.284076 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.284080 | controller |     - '        '
2026-03-14 09:44:27.284085 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.284089 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.284093 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.284097 | controller |     - '        '
2026-03-14 09:44:27.284101 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.284106 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.284110 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.284115 | controller |       available"'
2026-03-14 09:44:27.284121 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.284152 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.284160 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.284166 | controller |     - '          exit 1'
2026-03-14 09:44:27.284172 | controller |     - '        fi'
2026-03-14 09:44:27.284177 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.284183 | controller |     - '        '
2026-03-14 09:44:27.284188 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.284194 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.284199 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.284205 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.284221 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.284228 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.284233 | controller |       -- \'
2026-03-14 09:44:27.284239 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.284245 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.284251 | controller |     - '    logger.go:42: 09:28:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.284257 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.284262 | controller |     - '    logger.go:42: 09:28:55 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.284268 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.284274 | controller |       found'
2026-03-14 09:44:27.284279 | controller |     - '    logger.go:42: 09:28:56 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.284284 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.284310 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.284317 | controller |     - '        '
2026-03-14 09:44:27.284322 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.284328 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.284334 | controller |     - '        '
2026-03-14 09:44:27.284339 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.284345 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.284351 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.284357 | controller |     - '          exit 1'
2026-03-14 09:44:27.284375 | controller |     - '        fi'
2026-03-14 09:44:27.284382 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.284387 | controller |     - '        '
2026-03-14 09:44:27.284393 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.284399 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.284405 | controller |       | base64 -d)'
2026-03-14 09:44:27.284410 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.284419 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.284424 | controller |     - '          exit 1'
2026-03-14 09:44:27.284430 | controller |     - '        fi'
2026-03-14 09:44:27.284435 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.284441 | controller |     - '        '
2026-03-14 09:44:27.284447 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.284453 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.284458 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.284464 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.284469 | controller |       --timeout=300s'
2026-03-14 09:44:27.284475 | controller |     - '        '
2026-03-14 09:44:27.284480 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.284485 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.284490 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.284495 | controller |     - '        '
2026-03-14 09:44:27.284501 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.284506 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.284511 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.284516 | controller |     - '        '
2026-03-14 09:44:27.284521 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.284527 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.284532 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.284538 | controller |       available"'
2026-03-14 09:44:27.284544 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.284549 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.284555 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.284560 | controller |     - '          exit 1'
2026-03-14 09:44:27.284565 | controller |     - '        fi'
2026-03-14 09:44:27.284570 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.284575 | controller |     - '        '
2026-03-14 09:44:27.284581 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.284586 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.284592 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.284597 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.284624 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.284632 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.284641 | controller |       -- \'
2026-03-14 09:44:27.284646 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.284652 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.284657 | controller |     - '    logger.go:42: 09:28:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.284663 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.284668 | controller |     - '    logger.go:42: 09:28:56 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.284674 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.284679 | controller |       found'
2026-03-14 09:44:27.284685 | controller |     - '    logger.go:42: 09:28:57 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.284690 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.284707 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.284712 | controller |     - '        '
2026-03-14 09:44:27.284718 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.284724 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.284729 | controller |     - '        '
2026-03-14 09:44:27.284734 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.284740 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.284745 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.284751 | controller |     - '          exit 1'
2026-03-14 09:44:27.284766 | controller |     - '        fi'
2026-03-14 09:44:27.284772 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.284777 | controller |     - '        '
2026-03-14 09:44:27.284783 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.284788 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.284794 | controller |       | base64 -d)'
2026-03-14 09:44:27.284799 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.284805 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.284810 | controller |     - '          exit 1'
2026-03-14 09:44:27.284816 | controller |     - '        fi'
2026-03-14 09:44:27.284822 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.284828 | controller |     - '        '
2026-03-14 09:44:27.284833 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.284838 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.284844 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.284850 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.284856 | controller |       --timeout=300s'
2026-03-14 09:44:27.284866 | controller |     - '        '
2026-03-14 09:44:27.284872 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.284878 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.284883 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.284889 | controller |     - '        '
2026-03-14 09:44:27.284895 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.284900 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.284906 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.284923 | controller |     - '        '
2026-03-14 09:44:27.284929 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.284934 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.284940 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.284945 | controller |       available"'
2026-03-14 09:44:27.284951 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.284956 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.284962 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.284967 | controller |     - '          exit 1'
2026-03-14 09:44:27.284977 | controller |     - '        fi'
2026-03-14 09:44:27.284983 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.284988 | controller |     - '        '
2026-03-14 09:44:27.284994 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.284999 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.285005 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.285010 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.285016 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.285030 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.285036 | controller |       -- \'
2026-03-14 09:44:27.285042 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.285048 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.285054 | controller |     - '    logger.go:42: 09:28:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.285060 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.285065 | controller |     - '    logger.go:42: 09:28:57 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.285070 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.285074 | controller |       found'
2026-03-14 09:44:27.285078 | controller |     - '    logger.go:42: 09:28:58 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.285083 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.285087 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.285091 | controller |     - '        '
2026-03-14 09:44:27.285096 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.285100 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.285104 | controller |     - '        '
2026-03-14 09:44:27.285109 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.285113 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.285117 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.285121 | controller |     - '          exit 1'
2026-03-14 09:44:27.285126 | controller |     - '        fi'
2026-03-14 09:44:27.285130 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.285134 | controller |     - '        '
2026-03-14 09:44:27.285138 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.285142 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.285147 | controller |       | base64 -d)'
2026-03-14 09:44:27.285151 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.285165 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.285172 | controller |     - '          exit 1'
2026-03-14 09:44:27.285177 | controller |     - '        fi'
2026-03-14 09:44:27.285181 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.285185 | controller |     - '        '
2026-03-14 09:44:27.285189 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.285194 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.285198 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.285202 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.285206 | controller |       --timeout=300s'
2026-03-14 09:44:27.285211 | controller |     - '        '
2026-03-14 09:44:27.285215 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.285219 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.285223 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.285227 | controller |     - '        '
2026-03-14 09:44:27.285232 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.285236 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.285240 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.285244 | controller |     - '        '
2026-03-14 09:44:27.285248 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.285252 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.285257 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.285263 | controller |       available"'
2026-03-14 09:44:27.285268 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.285274 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.285279 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.285283 | controller |     - '          exit 1'
2026-03-14 09:44:27.285302 | controller |     - '        fi'
2026-03-14 09:44:27.285310 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.285314 | controller |     - '        '
2026-03-14 09:44:27.285318 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.285322 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.285327 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.285331 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.285335 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.285347 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.285353 | controller |       -- \'
2026-03-14 09:44:27.285359 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.285365 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.285370 | controller |     - '    logger.go:42: 09:28:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.285376 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.285391 | controller |     - '    logger.go:42: 09:28:58 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.285399 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.285405 | controller |       found'
2026-03-14 09:44:27.285410 | controller |     - '    logger.go:42: 09:28:59 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.285416 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.285420 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.285426 | controller |     - '        '
2026-03-14 09:44:27.285431 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.285437 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.285442 | controller |     - '        '
2026-03-14 09:44:27.285446 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.285450 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.285454 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.285459 | controller |     - '          exit 1'
2026-03-14 09:44:27.285463 | controller |     - '        fi'
2026-03-14 09:44:27.285467 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.285471 | controller |     - '        '
2026-03-14 09:44:27.285475 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.285480 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.285484 | controller |       | base64 -d)'
2026-03-14 09:44:27.285488 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.285492 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.285496 | controller |     - '          exit 1'
2026-03-14 09:44:27.285501 | controller |     - '        fi'
2026-03-14 09:44:27.285506 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.285511 | controller |     - '        '
2026-03-14 09:44:27.285517 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.285521 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.285525 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.285530 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.285534 | controller |       --timeout=300s'
2026-03-14 09:44:27.285538 | controller |     - '        '
2026-03-14 09:44:27.285542 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.285546 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.285550 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.285555 | controller |     - '        '
2026-03-14 09:44:27.285559 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.285563 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.285567 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.285571 | controller |     - '        '
2026-03-14 09:44:27.285576 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.285580 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.285585 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.285591 | controller |       available"'
2026-03-14 09:44:27.285597 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.285603 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.285611 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.285617 | controller |     - '          exit 1'
2026-03-14 09:44:27.285623 | controller |     - '        fi'
2026-03-14 09:44:27.285628 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.285634 | controller |     - '        '
2026-03-14 09:44:27.285639 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.285645 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.285651 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.285657 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.285663 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.285669 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.285675 | controller |       -- \'
2026-03-14 09:44:27.285690 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.285696 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.285701 | controller |     - '    logger.go:42: 09:28:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.285707 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.285712 | controller |     - '    logger.go:42: 09:28:59 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.285717 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.285721 | controller |       found'
2026-03-14 09:44:27.285725 | controller |     - '    logger.go:42: 09:29:00 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.285729 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.285733 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.285738 | controller |     - '        '
2026-03-14 09:44:27.285742 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.285746 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.285750 | controller |     - '        '
2026-03-14 09:44:27.285754 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.285758 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.285763 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.285767 | controller |     - '          exit 1'
2026-03-14 09:44:27.285771 | controller |     - '        fi'
2026-03-14 09:44:27.285775 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.285779 | controller |     - '        '
2026-03-14 09:44:27.285787 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.285791 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.285795 | controller |       | base64 -d)'
2026-03-14 09:44:27.285800 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.285804 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.285808 | controller |     - '          exit 1'
2026-03-14 09:44:27.285812 | controller |     - '        fi'
2026-03-14 09:44:27.285816 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.285821 | controller |     - '        '
2026-03-14 09:44:27.285825 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.285829 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.285833 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.285839 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.285844 | controller |       --timeout=300s'
2026-03-14 09:44:27.285848 | controller |     - '        '
2026-03-14 09:44:27.285852 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.285856 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.285860 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.285864 | controller |     - '        '
2026-03-14 09:44:27.285889 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.285895 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.285899 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.285904 | controller |     - '        '
2026-03-14 09:44:27.285908 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.285912 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.285916 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.285921 | controller |       available"'
2026-03-14 09:44:27.285925 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.285929 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.285933 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.285937 | controller |     - '          exit 1'
2026-03-14 09:44:27.285945 | controller |     - '        fi'
2026-03-14 09:44:27.285949 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.285954 | controller |     - '        '
2026-03-14 09:44:27.285958 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.285962 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.285966 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.285970 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.285975 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.285979 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.285983 | controller |       -- \'
2026-03-14 09:44:27.285994 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.286000 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.286006 | controller |     - '    logger.go:42: 09:29:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.286010 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.286014 | controller |     - '    logger.go:42: 09:29:01 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.286020 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.286033 | controller |       found'
2026-03-14 09:44:27.286038 | controller |     - '    logger.go:42: 09:29:02 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.286042 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.286046 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.286050 | controller |     - '        '
2026-03-14 09:44:27.286055 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.286061 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.286065 | controller |     - '        '
2026-03-14 09:44:27.286069 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.286074 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.286078 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.286082 | controller |     - '          exit 1'
2026-03-14 09:44:27.286086 | controller |     - '        fi'
2026-03-14 09:44:27.286091 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.286095 | controller |     - '        '
2026-03-14 09:44:27.286099 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.286103 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.286107 | controller |       | base64 -d)'
2026-03-14 09:44:27.286112 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.286116 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.286120 | controller |     - '          exit 1'
2026-03-14 09:44:27.286124 | controller |     - '        fi'
2026-03-14 09:44:27.286128 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.286133 | controller |     - '        '
2026-03-14 09:44:27.286137 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.286141 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.286147 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.286152 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.286158 | controller |       --timeout=300s'
2026-03-14 09:44:27.286164 | controller |     - '        '
2026-03-14 09:44:27.286170 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.286175 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.286181 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.286186 | controller |     - '        '
2026-03-14 09:44:27.286192 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.286198 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.286203 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.286209 | controller |     - '        '
2026-03-14 09:44:27.286215 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.286221 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.286226 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.286232 | controller |       available"'
2026-03-14 09:44:27.286237 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.286243 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.286249 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.286255 | controller |     - '          exit 1'
2026-03-14 09:44:27.286261 | controller |     - '        fi'
2026-03-14 09:44:27.286266 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.286272 | controller |     - '        '
2026-03-14 09:44:27.286277 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.286282 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.286314 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.286323 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.286327 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.286331 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.286336 | controller |       -- \'
2026-03-14 09:44:27.286347 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.286352 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.286356 | controller |     - '    logger.go:42: 09:29:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.286360 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.286364 | controller |     - '    logger.go:42: 09:29:02 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.286369 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.286373 | controller |       found'
2026-03-14 09:44:27.286377 | controller |     - '    logger.go:42: 09:29:03 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.286381 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.286385 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.286390 | controller |     - '        '
2026-03-14 09:44:27.286394 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.286398 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.286402 | controller |     - '        '
2026-03-14 09:44:27.286406 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.286410 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.286415 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.286419 | controller |     - '          exit 1'
2026-03-14 09:44:27.286423 | controller |     - '        fi'
2026-03-14 09:44:27.286427 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.286432 | controller |     - '        '
2026-03-14 09:44:27.286436 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.286440 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.286444 | controller |       | base64 -d)'
2026-03-14 09:44:27.286448 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.286452 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.286456 | controller |     - '          exit 1'
2026-03-14 09:44:27.286461 | controller |     - '        fi'
2026-03-14 09:44:27.286465 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.286469 | controller |     - '        '
2026-03-14 09:44:27.286473 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.286477 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.286482 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.286486 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.286490 | controller |       --timeout=300s'
2026-03-14 09:44:27.286494 | controller |     - '        '
2026-03-14 09:44:27.286498 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.286502 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.286507 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.286514 | controller |     - '        '
2026-03-14 09:44:27.286518 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.286522 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.286526 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.286530 | controller |     - '        '
2026-03-14 09:44:27.286535 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.286539 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.286543 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.286547 | controller |       available"'
2026-03-14 09:44:27.286551 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.286555 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.286560 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.286564 | controller |     - '          exit 1'
2026-03-14 09:44:27.286568 | controller |     - '        fi'
2026-03-14 09:44:27.286572 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.286576 | controller |     - '        '
2026-03-14 09:44:27.286581 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.286585 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.286589 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.286593 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.286597 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.286602 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.286606 | controller |       -- \'
2026-03-14 09:44:27.286610 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.286621 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.286628 | controller |     - '    logger.go:42: 09:29:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.286634 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.286640 | controller |     - '    logger.go:42: 09:29:03 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.286646 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.286652 | controller |       found'
2026-03-14 09:44:27.286661 | controller |     - '    logger.go:42: 09:29:04 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.286667 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.286672 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.286677 | controller |     - '        '
2026-03-14 09:44:27.286683 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.286689 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.286693 | controller |     - '        '
2026-03-14 09:44:27.286697 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.286701 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.286705 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.286709 | controller |     - '          exit 1'
2026-03-14 09:44:27.286714 | controller |     - '        fi'
2026-03-14 09:44:27.286718 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.286724 | controller |     - '        '
2026-03-14 09:44:27.286729 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.286733 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.286737 | controller |       | base64 -d)'
2026-03-14 09:44:27.286741 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.286745 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.286749 | controller |     - '          exit 1'
2026-03-14 09:44:27.286754 | controller |     - '        fi'
2026-03-14 09:44:27.286758 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.286762 | controller |     - '        '
2026-03-14 09:44:27.286766 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.286770 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.286774 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.286779 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.286783 | controller |       --timeout=300s'
2026-03-14 09:44:27.286787 | controller |     - '        '
2026-03-14 09:44:27.286791 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.286795 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.286800 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.286804 | controller |     - '        '
2026-03-14 09:44:27.286808 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.286812 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.286816 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.286820 | controller |     - '        '
2026-03-14 09:44:27.286825 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.286829 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.286833 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.286837 | controller |       available"'
2026-03-14 09:44:27.286841 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.286846 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.286850 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.286854 | controller |     - '          exit 1'
2026-03-14 09:44:27.286858 | controller |     - '        fi'
2026-03-14 09:44:27.286862 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.286866 | controller |     - '        '
2026-03-14 09:44:27.286871 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.286875 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.286879 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.286883 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.286887 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.286891 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.286896 | controller |       -- \'
2026-03-14 09:44:27.286901 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.286906 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.286917 | controller |     - '    logger.go:42: 09:29:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.286922 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.286926 | controller |     - '    logger.go:42: 09:29:04 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.286931 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.286935 | controller |       found'
2026-03-14 09:44:27.286939 | controller |     - '    logger.go:42: 09:29:05 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.286943 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.286947 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.286952 | controller |     - '        '
2026-03-14 09:44:27.286956 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.286960 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.286964 | controller |     - '        '
2026-03-14 09:44:27.286968 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.286972 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.286977 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.286981 | controller |     - '          exit 1'
2026-03-14 09:44:27.286985 | controller |     - '        fi'
2026-03-14 09:44:27.286989 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.286993 | controller |     - '        '
2026-03-14 09:44:27.286998 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.287002 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.287006 | controller |       | base64 -d)'
2026-03-14 09:44:27.287010 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.287014 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.287018 | controller |     - '          exit 1'
2026-03-14 09:44:27.287022 | controller |     - '        fi'
2026-03-14 09:44:27.287027 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.287031 | controller |     - '        '
2026-03-14 09:44:27.287035 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.287039 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.287043 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.287048 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.287052 | controller |       --timeout=300s'
2026-03-14 09:44:27.287065 | controller |     - '        '
2026-03-14 09:44:27.287070 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.287074 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.287078 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.287082 | controller |     - '        '
2026-03-14 09:44:27.287086 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.287091 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.287095 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.287099 | controller |     - '        '
2026-03-14 09:44:27.287103 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.287125 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.287130 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.287134 | controller |       available"'
2026-03-14 09:44:27.287138 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.287142 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.287146 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.287151 | controller |     - '          exit 1'
2026-03-14 09:44:27.287155 | controller |     - '        fi'
2026-03-14 09:44:27.287159 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.287165 | controller |     - '        '
2026-03-14 09:44:27.287169 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.287174 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.287178 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.287182 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.287186 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.287190 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.287195 | controller |       -- \'
2026-03-14 09:44:27.287199 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.287203 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.287213 | controller |     - '    logger.go:42: 09:29:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.287218 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.287222 | controller |     - '    logger.go:42: 09:29:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.287226 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.287230 | controller |       found'
2026-03-14 09:44:27.287237 | controller |     - '    logger.go:42: 09:29:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.287242 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.287246 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.287250 | controller |     - '        '
2026-03-14 09:44:27.287262 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.287266 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.287271 | controller |     - '        '
2026-03-14 09:44:27.287275 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.287279 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.287283 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.287290 | controller |     - '          exit 1'
2026-03-14 09:44:27.287309 | controller |     - '        fi'
2026-03-14 09:44:27.287314 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.287319 | controller |     - '        '
2026-03-14 09:44:27.287323 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.287327 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.287331 | controller |       | base64 -d)'
2026-03-14 09:44:27.287336 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.287340 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.287346 | controller |     - '          exit 1'
2026-03-14 09:44:27.287350 | controller |     - '        fi'
2026-03-14 09:44:27.287355 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.287359 | controller |     - '        '
2026-03-14 09:44:27.287363 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.287367 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.287371 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.287376 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.287380 | controller |       --timeout=300s'
2026-03-14 09:44:27.287384 | controller |     - '        '
2026-03-14 09:44:27.287388 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.287392 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.287397 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.287401 | controller |     - '        '
2026-03-14 09:44:27.287413 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.287418 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.287423 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.287427 | controller |     - '        '
2026-03-14 09:44:27.287431 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.287436 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.287440 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.287444 | controller |       available"'
2026-03-14 09:44:27.287448 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.287452 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.287457 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.287461 | controller |     - '          exit 1'
2026-03-14 09:44:27.287465 | controller |     - '        fi'
2026-03-14 09:44:27.287469 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.287473 | controller |     - '        '
2026-03-14 09:44:27.287477 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.287482 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.287486 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.287490 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.287494 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.287498 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.287503 | controller |       -- \'
2026-03-14 09:44:27.287507 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.287511 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.287515 | controller |     - '    logger.go:42: 09:29:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.287526 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.287530 | controller |     - '    logger.go:42: 09:29:06 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.287535 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.287541 | controller |       found'
2026-03-14 09:44:27.287545 | controller |     - '    logger.go:42: 09:29:07 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.287549 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.287553 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.287558 | controller |     - '        '
2026-03-14 09:44:27.287562 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.287566 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.287570 | controller |     - '        '
2026-03-14 09:44:27.287574 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.287579 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.287583 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.287587 | controller |     - '          exit 1'
2026-03-14 09:44:27.287591 | controller |     - '        fi'
2026-03-14 09:44:27.287596 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.287600 | controller |     - '        '
2026-03-14 09:44:27.287604 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.287608 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.287613 | controller |       | base64 -d)'
2026-03-14 09:44:27.287618 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.287624 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.287630 | controller |     - '          exit 1'
2026-03-14 09:44:27.287636 | controller |     - '        fi'
2026-03-14 09:44:27.287642 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.287648 | controller |     - '        '
2026-03-14 09:44:27.287654 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.287659 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.287665 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.287670 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.287676 | controller |       --timeout=300s'
2026-03-14 09:44:27.287681 | controller |     - '        '
2026-03-14 09:44:27.287686 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.287690 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.287695 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.287699 | controller |     - '        '
2026-03-14 09:44:27.287703 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.287710 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.287714 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.287718 | controller |     - '        '
2026-03-14 09:44:27.287722 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.287726 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.287731 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.287735 | controller |       available"'
2026-03-14 09:44:27.287739 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.287743 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.287747 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.287761 | controller |     - '          exit 1'
2026-03-14 09:44:27.287766 | controller |     - '        fi'
2026-03-14 09:44:27.287771 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.287775 | controller |     - '        '
2026-03-14 09:44:27.287779 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.287783 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.287787 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.287791 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.287796 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.287800 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.287804 | controller |       -- \'
2026-03-14 09:44:27.287808 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.287813 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.287817 | controller |     - '    logger.go:42: 09:29:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.287821 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.287832 | controller |     - '    logger.go:42: 09:29:07 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.287836 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.287840 | controller |       found'
2026-03-14 09:44:27.287845 | controller |     - '    logger.go:42: 09:29:08 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.287849 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.287853 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.287857 | controller |     - '        '
2026-03-14 09:44:27.287861 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.287876 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.287881 | controller |     - '        '
2026-03-14 09:44:27.287885 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.287889 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.287893 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.287898 | controller |     - '          exit 1'
2026-03-14 09:44:27.287902 | controller |     - '        fi'
2026-03-14 09:44:27.287907 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.287911 | controller |     - '        '
2026-03-14 09:44:27.287915 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.287919 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.287924 | controller |       | base64 -d)'
2026-03-14 09:44:27.287928 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.287932 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.287936 | controller |     - '          exit 1'
2026-03-14 09:44:27.287940 | controller |     - '        fi'
2026-03-14 09:44:27.287945 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.287949 | controller |     - '        '
2026-03-14 09:44:27.287953 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.287958 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.287963 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.287968 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.287974 | controller |       --timeout=300s'
2026-03-14 09:44:27.287980 | controller |     - '        '
2026-03-14 09:44:27.287985 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.287989 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.287994 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.287998 | controller |     - '        '
2026-03-14 09:44:27.288002 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.288006 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.288011 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.288015 | controller |     - '        '
2026-03-14 09:44:27.288019 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.288024 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.288030 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.288035 | controller |       available"'
2026-03-14 09:44:27.288041 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.288046 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.288052 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.288056 | controller |     - '          exit 1'
2026-03-14 09:44:27.288060 | controller |     - '        fi'
2026-03-14 09:44:27.288064 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.288068 | controller |     - '        '
2026-03-14 09:44:27.288073 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.288077 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.288081 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.288085 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.288089 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.288093 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.288098 | controller |       -- \'
2026-03-14 09:44:27.288102 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.288106 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.288110 | controller |     - '    logger.go:42: 09:29:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.288114 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.288125 | controller |     - '    logger.go:42: 09:29:09 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.288130 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.288134 | controller |       found'
2026-03-14 09:44:27.288138 | controller |     - '    logger.go:42: 09:29:10 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.288142 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.288147 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.288151 | controller |     - '        '
2026-03-14 09:44:27.288155 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.288161 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.288170 | controller |     - '        '
2026-03-14 09:44:27.288175 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.288181 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.288187 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.288193 | controller |     - '          exit 1'
2026-03-14 09:44:27.288199 | controller |     - '        fi'
2026-03-14 09:44:27.288205 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.288210 | controller |     - '        '
2026-03-14 09:44:27.288226 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.288231 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.288236 | controller |       | base64 -d)'
2026-03-14 09:44:27.288240 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.288244 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.288248 | controller |     - '          exit 1'
2026-03-14 09:44:27.288252 | controller |     - '        fi'
2026-03-14 09:44:27.288257 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.288263 | controller |     - '        '
2026-03-14 09:44:27.288276 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.288302 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.288310 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.288315 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.288319 | controller |       --timeout=300s'
2026-03-14 09:44:27.288323 | controller |     - '        '
2026-03-14 09:44:27.288328 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.288332 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.288348 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.288353 | controller |     - '        '
2026-03-14 09:44:27.288358 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.288362 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.288366 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.288370 | controller |     - '        '
2026-03-14 09:44:27.288375 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.288379 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.288383 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.288387 | controller |       available"'
2026-03-14 09:44:27.288391 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.288395 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.288400 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.288404 | controller |     - '          exit 1'
2026-03-14 09:44:27.288408 | controller |     - '        fi'
2026-03-14 09:44:27.288412 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.288416 | controller |     - '        '
2026-03-14 09:44:27.288420 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.288425 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.288429 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.288436 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.288440 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.288445 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.288450 | controller |       -- \'
2026-03-14 09:44:27.288456 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.288462 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.288467 | controller |     - '    logger.go:42: 09:29:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.288471 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.288476 | controller |     - '    logger.go:42: 09:29:10 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.288488 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.288493 | controller |       found'
2026-03-14 09:44:27.288497 | controller |     - '    logger.go:42: 09:29:11 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.288502 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.288506 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.288510 | controller |     - '        '
2026-03-14 09:44:27.288514 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.288519 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.288523 | controller |     - '        '
2026-03-14 09:44:27.288536 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.288542 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.288550 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.288554 | controller |     - '          exit 1'
2026-03-14 09:44:27.288558 | controller |     - '        fi'
2026-03-14 09:44:27.288563 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.288567 | controller |     - '        '
2026-03-14 09:44:27.288571 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.288584 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.288588 | controller |       | base64 -d)'
2026-03-14 09:44:27.288593 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.288597 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.288601 | controller |     - '          exit 1'
2026-03-14 09:44:27.288606 | controller |     - '        fi'
2026-03-14 09:44:27.288612 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.288617 | controller |     - '        '
2026-03-14 09:44:27.288623 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.288629 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.288634 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.288640 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.288646 | controller |       --timeout=300s'
2026-03-14 09:44:27.288651 | controller |     - '        '
2026-03-14 09:44:27.288656 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.288662 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.288668 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.288676 | controller |     - '        '
2026-03-14 09:44:27.288682 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.288688 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.288694 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.288700 | controller |     - '        '
2026-03-14 09:44:27.288706 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.288712 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.288717 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.288723 | controller |       available"'
2026-03-14 09:44:27.288728 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.288734 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.288739 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.288745 | controller |     - '          exit 1'
2026-03-14 09:44:27.288751 | controller |     - '        fi'
2026-03-14 09:44:27.288757 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.288762 | controller |     - '        '
2026-03-14 09:44:27.288768 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.288774 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.288779 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.288783 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.288787 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.288791 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.288796 | controller |       -- \'
2026-03-14 09:44:27.288800 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.288804 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.288808 | controller |     - '    logger.go:42: 09:29:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.288813 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.288817 | controller |     - '    logger.go:42: 09:29:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.288821 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.288834 | controller |       found'
2026-03-14 09:44:27.288838 | controller |     - '    logger.go:42: 09:29:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.288844 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.288850 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.288856 | controller |     - '        '
2026-03-14 09:44:27.288860 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.288864 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.288869 | controller |     - '        '
2026-03-14 09:44:27.288873 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.288877 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.288881 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.288885 | controller |     - '          exit 1'
2026-03-14 09:44:27.288890 | controller |     - '        fi'
2026-03-14 09:44:27.288894 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.288898 | controller |     - '        '
2026-03-14 09:44:27.288905 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.288911 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.288916 | controller |       | base64 -d)'
2026-03-14 09:44:27.288922 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.288931 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.288936 | controller |     - '          exit 1'
2026-03-14 09:44:27.288940 | controller |     - '        fi'
2026-03-14 09:44:27.288945 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.288949 | controller |     - '        '
2026-03-14 09:44:27.288953 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.288957 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.288961 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.288966 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.288970 | controller |       --timeout=300s'
2026-03-14 09:44:27.288974 | controller |     - '        '
2026-03-14 09:44:27.288978 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.288983 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.288987 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.288991 | controller |     - '        '
2026-03-14 09:44:27.288995 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.289000 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.289006 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.289012 | controller |     - '        '
2026-03-14 09:44:27.289016 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.289020 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.289024 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.289029 | controller |       available"'
2026-03-14 09:44:27.289033 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.289037 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.289041 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.289045 | controller |     - '          exit 1'
2026-03-14 09:44:27.289050 | controller |     - '        fi'
2026-03-14 09:44:27.289054 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.289058 | controller |     - '        '
2026-03-14 09:44:27.289062 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.289066 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.289071 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.289075 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.289080 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.289086 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.289091 | controller |       -- \'
2026-03-14 09:44:27.289095 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.289102 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.289106 | controller |     - '    logger.go:42: 09:29:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.289110 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.289114 | controller |     - '    logger.go:42: 09:29:12 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.289119 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.289123 | controller |       found'
2026-03-14 09:44:27.289134 | controller |     - '    logger.go:42: 09:29:13 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.289138 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.289143 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.289147 | controller |     - '        '
2026-03-14 09:44:27.289151 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.289156 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.289162 | controller |     - '        '
2026-03-14 09:44:27.289167 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.289172 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.289176 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.289180 | controller |     - '          exit 1'
2026-03-14 09:44:27.289184 | controller |     - '        fi'
2026-03-14 09:44:27.289189 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.289204 | controller |     - '        '
2026-03-14 09:44:27.289209 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.289213 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.289217 | controller |       | base64 -d)'
2026-03-14 09:44:27.289221 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.289225 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.289230 | controller |     - '          exit 1'
2026-03-14 09:44:27.289235 | controller |     - '        fi'
2026-03-14 09:44:27.289241 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.289246 | controller |     - '        '
2026-03-14 09:44:27.289250 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.289255 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.289259 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.289263 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.289267 | controller |       --timeout=300s'
2026-03-14 09:44:27.289281 | controller |     - '        '
2026-03-14 09:44:27.289286 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.289305 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.289312 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.289318 | controller |     - '        '
2026-03-14 09:44:27.289324 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.289328 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.289332 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.289336 | controller |     - '        '
2026-03-14 09:44:27.289341 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.289355 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.289360 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.289364 | controller |       available"'
2026-03-14 09:44:27.289368 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.289372 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.289377 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.289381 | controller |     - '          exit 1'
2026-03-14 09:44:27.289385 | controller |     - '        fi'
2026-03-14 09:44:27.289389 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.289395 | controller |     - '        '
2026-03-14 09:44:27.289401 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.289406 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.289410 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.289416 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.289425 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.289431 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.289437 | controller |       -- \'
2026-03-14 09:44:27.289443 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.289449 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.289454 | controller |     - '    logger.go:42: 09:29:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.289458 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.289466 | controller |     - '    logger.go:42: 09:29:13 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.289472 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.289478 | controller |       found'
2026-03-14 09:44:27.289484 | controller |     - '    logger.go:42: 09:29:14 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.289499 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.289506 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.289512 | controller |     - '        '
2026-03-14 09:44:27.289518 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.289522 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.289526 | controller |     - '        '
2026-03-14 09:44:27.289530 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.289534 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.289539 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.289543 | controller |     - '          exit 1'
2026-03-14 09:44:27.289547 | controller |     - '        fi'
2026-03-14 09:44:27.289551 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.289555 | controller |     - '        '
2026-03-14 09:44:27.289560 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.289564 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.289568 | controller |       | base64 -d)'
2026-03-14 09:44:27.289572 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.289576 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.289580 | controller |     - '          exit 1'
2026-03-14 09:44:27.289585 | controller |     - '        fi'
2026-03-14 09:44:27.289591 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.289596 | controller |     - '        '
2026-03-14 09:44:27.289600 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.289604 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.289608 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.289613 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.289619 | controller |       --timeout=300s'
2026-03-14 09:44:27.289625 | controller |     - '        '
2026-03-14 09:44:27.289630 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.289634 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.289639 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.289643 | controller |     - '        '
2026-03-14 09:44:27.289647 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.289651 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.289655 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.289660 | controller |     - '        '
2026-03-14 09:44:27.289664 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.289668 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.289672 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.289676 | controller |       available"'
2026-03-14 09:44:27.289680 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.289685 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.289689 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.289693 | controller |     - '          exit 1'
2026-03-14 09:44:27.289697 | controller |     - '        fi'
2026-03-14 09:44:27.289702 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.289706 | controller |     - '        '
2026-03-14 09:44:27.289712 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.289717 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.289722 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.289727 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.289731 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.289735 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.289739 | controller |       -- \'
2026-03-14 09:44:27.289743 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.289748 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.289752 | controller |     - '    logger.go:42: 09:29:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.289756 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.289760 | controller |     - '    logger.go:42: 09:29:14 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.289764 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.289779 | controller |       found'
2026-03-14 09:44:27.289786 | controller |     - '    logger.go:42: 09:29:15 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.289791 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.289796 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.289806 | controller |     - '        '
2026-03-14 09:44:27.289811 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.289815 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.289819 | controller |     - '        '
2026-03-14 09:44:27.289823 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.289827 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.289832 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.289836 | controller |     - '          exit 1'
2026-03-14 09:44:27.289842 | controller |     - '        fi'
2026-03-14 09:44:27.289848 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.289853 | controller |     - '        '
2026-03-14 09:44:27.289857 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.289862 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.289867 | controller |       | base64 -d)'
2026-03-14 09:44:27.289873 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.289893 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.289900 | controller |     - '          exit 1'
2026-03-14 09:44:27.289906 | controller |     - '        fi'
2026-03-14 09:44:27.289911 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.289917 | controller |     - '        '
2026-03-14 09:44:27.289923 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.289929 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.289938 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.289944 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.289948 | controller |       --timeout=300s'
2026-03-14 09:44:27.289953 | controller |     - '        '
2026-03-14 09:44:27.289957 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.289961 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.289965 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.289969 | controller |     - '        '
2026-03-14 09:44:27.289974 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.289978 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.289982 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.289986 | controller |     - '        '
2026-03-14 09:44:27.289990 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.289995 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.289999 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.290003 | controller |       available"'
2026-03-14 09:44:27.290007 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.290021 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.290025 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.290036 | controller |     - '          exit 1'
2026-03-14 09:44:27.290041 | controller |     - '        fi'
2026-03-14 09:44:27.290045 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.290049 | controller |     - '        '
2026-03-14 09:44:27.290053 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.290057 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.290062 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.290067 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.290073 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.290078 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.290084 | controller |       -- \'
2026-03-14 09:44:27.290089 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.290094 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.290100 | controller |     - '    logger.go:42: 09:29:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.290104 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.290108 | controller |     - '    logger.go:42: 09:29:15 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.290113 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.290117 | controller |       found'
2026-03-14 09:44:27.290121 | controller |     - '    logger.go:42: 09:29:16 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.290125 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.290129 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.290134 | controller |     - '        '
2026-03-14 09:44:27.290145 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.290149 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.290153 | controller |     - '        '
2026-03-14 09:44:27.290158 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.290162 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.290166 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.290170 | controller |     - '          exit 1'
2026-03-14 09:44:27.290175 | controller |     - '        fi'
2026-03-14 09:44:27.290179 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.290183 | controller |     - '        '
2026-03-14 09:44:27.290187 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.290191 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.290195 | controller |       | base64 -d)'
2026-03-14 09:44:27.290200 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.290204 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.290208 | controller |     - '          exit 1'
2026-03-14 09:44:27.290212 | controller |     - '        fi'
2026-03-14 09:44:27.290216 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.290221 | controller |     - '        '
2026-03-14 09:44:27.290225 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.290229 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.290233 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.290237 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.290244 | controller |       --timeout=300s'
2026-03-14 09:44:27.290248 | controller |     - '        '
2026-03-14 09:44:27.290253 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.290257 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.290261 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.290265 | controller |     - '        '
2026-03-14 09:44:27.290269 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.290273 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.290278 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.290282 | controller |     - '        '
2026-03-14 09:44:27.290286 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.290305 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.290310 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.290314 | controller |       available"'
2026-03-14 09:44:27.290318 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.290322 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.290326 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.290331 | controller |     - '          exit 1'
2026-03-14 09:44:27.290335 | controller |     - '        fi'
2026-03-14 09:44:27.290339 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.290343 | controller |     - '        '
2026-03-14 09:44:27.290347 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.290351 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.290358 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.290362 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.290367 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.290371 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.290375 | controller |       -- \'
2026-03-14 09:44:27.290379 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.290383 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.290396 | controller |     - '    logger.go:42: 09:29:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.290400 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.290405 | controller |     - '    logger.go:42: 09:29:17 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.290409 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.290413 | controller |       found'
2026-03-14 09:44:27.290417 | controller |     - '    logger.go:42: 09:29:18 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.290421 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.290426 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.290430 | controller |     - '        '
2026-03-14 09:44:27.290434 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.290444 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.290449 | controller |     - '        '
2026-03-14 09:44:27.290460 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.290465 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.290469 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.290473 | controller |     - '          exit 1'
2026-03-14 09:44:27.290477 | controller |     - '        fi'
2026-03-14 09:44:27.290482 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.290486 | controller |     - '        '
2026-03-14 09:44:27.290490 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.290494 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.290499 | controller |       | base64 -d)'
2026-03-14 09:44:27.290503 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.290507 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.290511 | controller |     - '          exit 1'
2026-03-14 09:44:27.290516 | controller |     - '        fi'
2026-03-14 09:44:27.290520 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.290524 | controller |     - '        '
2026-03-14 09:44:27.290528 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.290533 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.290537 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.290543 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.290548 | controller |       --timeout=300s'
2026-03-14 09:44:27.290552 | controller |     - '        '
2026-03-14 09:44:27.290556 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.290561 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.290565 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.290569 | controller |     - '        '
2026-03-14 09:44:27.290576 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.290580 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.290586 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.290590 | controller |     - '        '
2026-03-14 09:44:27.290594 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.290599 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.290603 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.290607 | controller |       available"'
2026-03-14 09:44:27.290611 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.290616 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.290620 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.290624 | controller |     - '          exit 1'
2026-03-14 09:44:27.290628 | controller |     - '        fi'
2026-03-14 09:44:27.290633 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.290637 | controller |     - '        '
2026-03-14 09:44:27.290641 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.290645 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.290649 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.290657 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.290661 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.290665 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.290669 | controller |       -- \'
2026-03-14 09:44:27.290674 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.290678 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.290682 | controller |     - '    logger.go:42: 09:29:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.290686 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.290690 | controller |     - '    logger.go:42: 09:29:18 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.290695 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.290699 | controller |       found'
2026-03-14 09:44:27.290703 | controller |     - '    logger.go:42: 09:29:19 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.290707 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.290712 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.290716 | controller |     - '        '
2026-03-14 09:44:27.290720 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.290730 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.290735 | controller |     - '        '
2026-03-14 09:44:27.290739 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.290743 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.290747 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.290752 | controller |     - '          exit 1'
2026-03-14 09:44:27.290756 | controller |     - '        fi'
2026-03-14 09:44:27.290760 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.290764 | controller |     - '        '
2026-03-14 09:44:27.290769 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.290773 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.290777 | controller |       | base64 -d)'
2026-03-14 09:44:27.290781 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.290785 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.290790 | controller |     - '          exit 1'
2026-03-14 09:44:27.290794 | controller |     - '        fi'
2026-03-14 09:44:27.290798 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.290802 | controller |     - '        '
2026-03-14 09:44:27.290807 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.290811 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.290815 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.290819 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.290823 | controller |       --timeout=300s'
2026-03-14 09:44:27.290827 | controller |     - '        '
2026-03-14 09:44:27.290832 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.290836 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.290840 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.290844 | controller |     - '        '
2026-03-14 09:44:27.290848 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.290854 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.290859 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.290863 | controller |     - '        '
2026-03-14 09:44:27.290867 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.290871 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.290876 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.290880 | controller |       available"'
2026-03-14 09:44:27.290884 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.290888 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.290892 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.290896 | controller |     - '          exit 1'
2026-03-14 09:44:27.290901 | controller |     - '        fi'
2026-03-14 09:44:27.290905 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.290909 | controller |     - '        '
2026-03-14 09:44:27.290913 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.290917 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.290922 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.290926 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.290930 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.290934 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.290938 | controller |       -- \'
2026-03-14 09:44:27.290942 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.290947 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.290951 | controller |     - '    logger.go:42: 09:29:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.290955 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.290959 | controller |     - '    logger.go:42: 09:29:19 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.290963 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.290968 | controller |       found'
2026-03-14 09:44:27.290972 | controller |     - '    logger.go:42: 09:29:20 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.290976 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.290980 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.290984 | controller |     - '        '
2026-03-14 09:44:27.290989 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.290993 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.290997 | controller |     - '        '
2026-03-14 09:44:27.291006 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.291013 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.291018 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.291024 | controller |     - '          exit 1'
2026-03-14 09:44:27.291028 | controller |     - '        fi'
2026-03-14 09:44:27.291032 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.291037 | controller |     - '        '
2026-03-14 09:44:27.291041 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.291053 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.291058 | controller |       | base64 -d)'
2026-03-14 09:44:27.291062 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.291066 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.291070 | controller |     - '          exit 1'
2026-03-14 09:44:27.291075 | controller |     - '        fi'
2026-03-14 09:44:27.291079 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.291083 | controller |     - '        '
2026-03-14 09:44:27.291087 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.291091 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.291096 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.291100 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.291104 | controller |       --timeout=300s'
2026-03-14 09:44:27.291108 | controller |     - '        '
2026-03-14 09:44:27.291112 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.291117 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.291121 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.291125 | controller |     - '        '
2026-03-14 09:44:27.291129 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.291134 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.291138 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.291142 | controller |     - '        '
2026-03-14 09:44:27.291153 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.291158 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.291163 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.291167 | controller |       available"'
2026-03-14 09:44:27.291171 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.291176 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.291180 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.291184 | controller |     - '          exit 1'
2026-03-14 09:44:27.291188 | controller |     - '        fi'
2026-03-14 09:44:27.291192 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.291197 | controller |     - '        '
2026-03-14 09:44:27.291201 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.291205 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.291209 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.291213 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.291217 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.291222 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.291226 | controller |       -- \'
2026-03-14 09:44:27.291230 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.291241 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.291246 | controller |     - '    logger.go:42: 09:29:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.291250 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.291254 | controller |     - '    logger.go:42: 09:29:20 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.291259 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.291264 | controller |       found'
2026-03-14 09:44:27.291268 | controller |     - '    logger.go:42: 09:29:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.291272 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.291276 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.291280 | controller |     - '        '
2026-03-14 09:44:27.291285 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.291302 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.291309 | controller |     - '        '
2026-03-14 09:44:27.291322 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.291333 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.291347 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.291354 | controller |     - '          exit 1'
2026-03-14 09:44:27.291360 | controller |     - '        fi'
2026-03-14 09:44:27.291364 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.291369 | controller |     - '        '
2026-03-14 09:44:27.291377 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.291381 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.291385 | controller |       | base64 -d)'
2026-03-14 09:44:27.291389 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.291394 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.291398 | controller |     - '          exit 1'
2026-03-14 09:44:27.291403 | controller |     - '        fi'
2026-03-14 09:44:27.291407 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.291411 | controller |     - '        '
2026-03-14 09:44:27.291416 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.291420 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.291424 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.291428 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.291432 | controller |       --timeout=300s'
2026-03-14 09:44:27.291437 | controller |     - '        '
2026-03-14 09:44:27.291441 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.291445 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.291449 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.291453 | controller |     - '        '
2026-03-14 09:44:27.291457 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.291462 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.291466 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.291470 | controller |     - '        '
2026-03-14 09:44:27.291474 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.291478 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.291495 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.291499 | controller |       available"'
2026-03-14 09:44:27.291504 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.291508 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.291512 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.291516 | controller |     - '          exit 1'
2026-03-14 09:44:27.291520 | controller |     - '        fi'
2026-03-14 09:44:27.291525 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.291529 | controller |     - '        '
2026-03-14 09:44:27.291533 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.291537 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.291541 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.291545 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.291550 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.291554 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.291561 | controller |       -- \'
2026-03-14 09:44:27.291565 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.291570 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.291574 | controller |     - '    logger.go:42: 09:29:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.291578 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.291582 | controller |     - '    logger.go:42: 09:29:21 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.291586 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.291591 | controller |       found'
2026-03-14 09:44:27.291595 | controller |     - '    logger.go:42: 09:29:22 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.291599 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.291603 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.291607 | controller |     - '        '
2026-03-14 09:44:27.291612 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.291616 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.291620 | controller |     - '        '
2026-03-14 09:44:27.291624 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.291637 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.291644 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.291651 | controller |     - '          exit 1'
2026-03-14 09:44:27.291658 | controller |     - '        fi'
2026-03-14 09:44:27.291664 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.291669 | controller |     - '        '
2026-03-14 09:44:27.291673 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.291678 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.291682 | controller |       | base64 -d)'
2026-03-14 09:44:27.291686 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.291691 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.291695 | controller |     - '          exit 1'
2026-03-14 09:44:27.291703 | controller |     - '        fi'
2026-03-14 09:44:27.291708 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.291724 | controller |     - '        '
2026-03-14 09:44:27.291729 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.291733 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.291738 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.291742 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.291746 | controller |       --timeout=300s'
2026-03-14 09:44:27.291751 | controller |     - '        '
2026-03-14 09:44:27.291755 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.291759 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.291764 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.291768 | controller |     - '        '
2026-03-14 09:44:27.291774 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.291779 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.291783 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.291787 | controller |     - '        '
2026-03-14 09:44:27.291792 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.291796 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.291802 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.291806 | controller |       available"'
2026-03-14 09:44:27.291811 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.291815 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.291819 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.291823 | controller |     - '          exit 1'
2026-03-14 09:44:27.291828 | controller |     - '        fi'
2026-03-14 09:44:27.291832 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.291836 | controller |     - '        '
2026-03-14 09:44:27.291840 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.291845 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.291849 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.291853 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.291858 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.291862 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.291866 | controller |       -- \'
2026-03-14 09:44:27.291871 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.291876 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.291880 | controller |     - '    logger.go:42: 09:29:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.291884 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.291889 | controller |     - '    logger.go:42: 09:29:22 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.291893 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.291897 | controller |       found'
2026-03-14 09:44:27.291904 | controller |     - '    logger.go:42: 09:29:23 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.291908 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.291913 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.291918 | controller |     - '        '
2026-03-14 09:44:27.291923 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.291928 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.291933 | controller |     - '        '
2026-03-14 09:44:27.291938 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.291943 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.291954 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.291961 | controller |     - '          exit 1'
2026-03-14 09:44:27.291967 | controller |     - '        fi'
2026-03-14 09:44:27.291974 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.291980 | controller |     - '        '
2026-03-14 09:44:27.291986 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.291991 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.291995 | controller |       | base64 -d)'
2026-03-14 09:44:27.291999 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.292004 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.292008 | controller |     - '          exit 1'
2026-03-14 09:44:27.292013 | controller |     - '        fi'
2026-03-14 09:44:27.292017 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.292021 | controller |     - '        '
2026-03-14 09:44:27.292025 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.292030 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.292034 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.292038 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.292042 | controller |       --timeout=300s'
2026-03-14 09:44:27.292046 | controller |     - '        '
2026-03-14 09:44:27.292051 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.292055 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.292059 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.292063 | controller |     - '        '
2026-03-14 09:44:27.292067 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.292072 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.292076 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.292080 | controller |     - '        '
2026-03-14 09:44:27.292084 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.292088 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.292093 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.292097 | controller |       available"'
2026-03-14 09:44:27.292101 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.292105 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.292109 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.292113 | controller |     - '          exit 1'
2026-03-14 09:44:27.292118 | controller |     - '        fi'
2026-03-14 09:44:27.292125 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.292129 | controller |     - '        '
2026-03-14 09:44:27.292133 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.292137 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.292141 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.292146 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.292150 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.292154 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.292158 | controller |       -- \'
2026-03-14 09:44:27.292162 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.292167 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.292171 | controller |     - '    logger.go:42: 09:29:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.292175 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.292179 | controller |     - '    logger.go:42: 09:29:23 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.292183 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.292187 | controller |       found'
2026-03-14 09:44:27.292192 | controller |     - '    logger.go:42: 09:29:24 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.292196 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.292200 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.292204 | controller |     - '        '
2026-03-14 09:44:27.292208 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.292213 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.292217 | controller |     - '        '
2026-03-14 09:44:27.292221 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.292225 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.292229 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.292240 | controller |     - '          exit 1'
2026-03-14 09:44:27.292247 | controller |     - '        fi'
2026-03-14 09:44:27.292253 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.292259 | controller |     - '        '
2026-03-14 09:44:27.292265 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.292270 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.292274 | controller |       | base64 -d)'
2026-03-14 09:44:27.292278 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.292282 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.292290 | controller |     - '          exit 1'
2026-03-14 09:44:27.292310 | controller |     - '        fi'
2026-03-14 09:44:27.292314 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.292319 | controller |     - '        '
2026-03-14 09:44:27.292323 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.292327 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.292331 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.292335 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.292340 | controller |       --timeout=300s'
2026-03-14 09:44:27.292344 | controller |     - '        '
2026-03-14 09:44:27.292358 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.292363 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.292367 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.292371 | controller |     - '        '
2026-03-14 09:44:27.292375 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.292380 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.292384 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.292388 | controller |     - '        '
2026-03-14 09:44:27.292392 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.292396 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.292401 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.292405 | controller |       available"'
2026-03-14 09:44:27.292409 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.292417 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.292421 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.292425 | controller |     - '          exit 1'
2026-03-14 09:44:27.292430 | controller |     - '        fi'
2026-03-14 09:44:27.292434 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.292438 | controller |     - '        '
2026-03-14 09:44:27.292442 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.292446 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.292450 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.292455 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.292459 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.292463 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.292467 | controller |       -- \'
2026-03-14 09:44:27.292471 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.292476 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.292480 | controller |     - '    logger.go:42: 09:29:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.292484 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.292488 | controller |     - '    logger.go:42: 09:29:25 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.292492 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.292497 | controller |       found'
2026-03-14 09:44:27.292501 | controller |     - '    logger.go:42: 09:29:26 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.292505 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.292511 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.292515 | controller |     - '        '
2026-03-14 09:44:27.292520 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.292524 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.292528 | controller |     - '        '
2026-03-14 09:44:27.292532 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.292544 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.292549 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.292553 | controller |     - '          exit 1'
2026-03-14 09:44:27.292557 | controller |     - '        fi'
2026-03-14 09:44:27.292568 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.292575 | controller |     - '        '
2026-03-14 09:44:27.292581 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.292588 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.292594 | controller |       | base64 -d)'
2026-03-14 09:44:27.292599 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.292603 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.292607 | controller |     - '          exit 1'
2026-03-14 09:44:27.292612 | controller |     - '        fi'
2026-03-14 09:44:27.292616 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.292620 | controller |     - '        '
2026-03-14 09:44:27.292624 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.292629 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.292633 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.292637 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.292641 | controller |       --timeout=300s'
2026-03-14 09:44:27.292645 | controller |     - '        '
2026-03-14 09:44:27.292650 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.292654 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.292658 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.292662 | controller |     - '        '
2026-03-14 09:44:27.292666 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.292671 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.292678 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.292683 | controller |     - '        '
2026-03-14 09:44:27.292687 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.292691 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.292695 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.292699 | controller |       available"'
2026-03-14 09:44:27.292703 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.292709 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.292713 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.292718 | controller |     - '          exit 1'
2026-03-14 09:44:27.292722 | controller |     - '        fi'
2026-03-14 09:44:27.292726 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.292730 | controller |     - '        '
2026-03-14 09:44:27.292734 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.292739 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.292743 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.292747 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.292753 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.292757 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.292762 | controller |       -- \'
2026-03-14 09:44:27.292766 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.292770 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.292774 | controller |     - '    logger.go:42: 09:29:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.292778 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.292783 | controller |     - '    logger.go:42: 09:29:26 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.292787 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.292791 | controller |       found'
2026-03-14 09:44:27.292795 | controller |     - '    logger.go:42: 09:29:27 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.292800 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.292805 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.292811 | controller |     - '        '
2026-03-14 09:44:27.292827 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.292832 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.292837 | controller |     - '        '
2026-03-14 09:44:27.292843 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.292847 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.292851 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.292856 | controller |     - '          exit 1'
2026-03-14 09:44:27.292860 | controller |     - '        fi'
2026-03-14 09:44:27.292864 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.292875 | controller |     - '        '
2026-03-14 09:44:27.292882 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.292888 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.292894 | controller |       | base64 -d)'
2026-03-14 09:44:27.292900 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.292905 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.292910 | controller |     - '          exit 1'
2026-03-14 09:44:27.292914 | controller |     - '        fi'
2026-03-14 09:44:27.292918 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.292922 | controller |     - '        '
2026-03-14 09:44:27.292926 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.292931 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.292935 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.292939 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.292943 | controller |       --timeout=300s'
2026-03-14 09:44:27.292947 | controller |     - '        '
2026-03-14 09:44:27.292951 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.292956 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.292960 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.292964 | controller |     - '        '
2026-03-14 09:44:27.292968 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.292975 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.292979 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.292983 | controller |     - '        '
2026-03-14 09:44:27.292988 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.292992 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.292996 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.293000 | controller |       available"'
2026-03-14 09:44:27.293005 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.293009 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.293013 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.293017 | controller |     - '          exit 1'
2026-03-14 09:44:27.293021 | controller |     - '        fi'
2026-03-14 09:44:27.293025 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.293030 | controller |     - '        '
2026-03-14 09:44:27.293036 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.293041 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.293045 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.293049 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.293055 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.293061 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.293067 | controller |       -- \'
2026-03-14 09:44:27.293071 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.293075 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.293079 | controller |     - '    logger.go:42: 09:29:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.293084 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.293088 | controller |     - '    logger.go:42: 09:29:27 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.293092 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.293096 | controller |       found'
2026-03-14 09:44:27.293100 | controller |     - '    logger.go:42: 09:29:28 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.293105 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.293109 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.293113 | controller |     - '        '
2026-03-14 09:44:27.293117 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.293121 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.293126 | controller |     - '        '
2026-03-14 09:44:27.293130 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.293134 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.293138 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.293142 | controller |     - '          exit 1'
2026-03-14 09:44:27.293147 | controller |     - '        fi'
2026-03-14 09:44:27.293151 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.293155 | controller |     - '        '
2026-03-14 09:44:27.293166 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.293173 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.293182 | controller |       | base64 -d)'
2026-03-14 09:44:27.293188 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.293196 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.293201 | controller |     - '          exit 1'
2026-03-14 09:44:27.293205 | controller |     - '        fi'
2026-03-14 09:44:27.293209 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.293213 | controller |     - '        '
2026-03-14 09:44:27.293218 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.293222 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.293226 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.293230 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.293234 | controller |       --timeout=300s'
2026-03-14 09:44:27.293238 | controller |     - '        '
2026-03-14 09:44:27.293243 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.293247 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.293251 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.293255 | controller |     - '        '
2026-03-14 09:44:27.293259 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.293263 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.293268 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.293272 | controller |     - '        '
2026-03-14 09:44:27.293276 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.293280 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.293284 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.293303 | controller |       available"'
2026-03-14 09:44:27.293308 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.293312 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.293316 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.293321 | controller |     - '          exit 1'
2026-03-14 09:44:27.293325 | controller |     - '        fi'
2026-03-14 09:44:27.293329 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.293333 | controller |     - '        '
2026-03-14 09:44:27.293337 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.293342 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.293346 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.293350 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.293354 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.293358 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.293362 | controller |       -- \'
2026-03-14 09:44:27.293367 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.293371 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.293378 | controller |     - '    logger.go:42: 09:29:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.293382 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.293386 | controller |     - '    logger.go:42: 09:29:28 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.293390 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.293394 | controller |       found'
2026-03-14 09:44:27.293399 | controller |     - '    logger.go:42: 09:29:29 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.293403 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.293407 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.293411 | controller |     - '        '
2026-03-14 09:44:27.293415 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.293420 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.293424 | controller |     - '        '
2026-03-14 09:44:27.293428 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.293432 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.293436 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.293440 | controller |     - '          exit 1'
2026-03-14 09:44:27.293445 | controller |     - '        fi'
2026-03-14 09:44:27.293449 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.293453 | controller |     - '        '
2026-03-14 09:44:27.293457 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.293469 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.293476 | controller |       | base64 -d)'
2026-03-14 09:44:27.293482 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.293488 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.293494 | controller |     - '          exit 1'
2026-03-14 09:44:27.293499 | controller |     - '        fi'
2026-03-14 09:44:27.293504 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.293508 | controller |     - '        '
2026-03-14 09:44:27.293512 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.293516 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.293520 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.293525 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.293529 | controller |       --timeout=300s'
2026-03-14 09:44:27.293536 | controller |     - '        '
2026-03-14 09:44:27.293541 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.293545 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.293549 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.293553 | controller |     - '        '
2026-03-14 09:44:27.293558 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.293562 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.293566 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.293570 | controller |     - '        '
2026-03-14 09:44:27.293574 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.293579 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.293583 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.293589 | controller |       available"'
2026-03-14 09:44:27.293593 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.293598 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.293602 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.293606 | controller |     - '          exit 1'
2026-03-14 09:44:27.293610 | controller |     - '        fi'
2026-03-14 09:44:27.293614 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.293619 | controller |     - '        '
2026-03-14 09:44:27.293623 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.293627 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.293631 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.293635 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.293640 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.293644 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.293648 | controller |       -- \'
2026-03-14 09:44:27.293652 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.293657 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.293672 | controller |     - '    logger.go:42: 09:29:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.293676 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.293681 | controller |     - '    logger.go:42: 09:29:29 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.293685 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.293689 | controller |       found'
2026-03-14 09:44:27.293693 | controller |     - '    logger.go:42: 09:29:30 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.293698 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.293702 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.293706 | controller |     - '        '
2026-03-14 09:44:27.293710 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.293714 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.293718 | controller |     - '        '
2026-03-14 09:44:27.293723 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.293727 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.293731 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.293735 | controller |     - '          exit 1'
2026-03-14 09:44:27.293740 | controller |     - '        fi'
2026-03-14 09:44:27.293744 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.293748 | controller |     - '        '
2026-03-14 09:44:27.293752 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.293764 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.293770 | controller |       | base64 -d)'
2026-03-14 09:44:27.293776 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.293782 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.293791 | controller |     - '          exit 1'
2026-03-14 09:44:27.293798 | controller |     - '        fi'
2026-03-14 09:44:27.293803 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.293809 | controller |     - '        '
2026-03-14 09:44:27.293818 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.293824 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.293830 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.293835 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.293841 | controller |       --timeout=300s'
2026-03-14 09:44:27.293847 | controller |     - '        '
2026-03-14 09:44:27.293852 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.293856 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.293861 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.293865 | controller |     - '        '
2026-03-14 09:44:27.293869 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.293873 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.293877 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.293882 | controller |     - '        '
2026-03-14 09:44:27.293902 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.293908 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.293913 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.293917 | controller |       available"'
2026-03-14 09:44:27.293924 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.293928 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.293932 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.293937 | controller |     - '          exit 1'
2026-03-14 09:44:27.293941 | controller |     - '        fi'
2026-03-14 09:44:27.293945 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.293949 | controller |     - '        '
2026-03-14 09:44:27.293953 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.293957 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.293962 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.293966 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.293970 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.293974 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.293978 | controller |       -- \'
2026-03-14 09:44:27.293983 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.293987 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.293991 | controller |     - '    logger.go:42: 09:29:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.293995 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.293999 | controller |     - '    logger.go:42: 09:29:30 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.294003 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.294008 | controller |       found'
2026-03-14 09:44:27.294012 | controller |     - '    logger.go:42: 09:29:31 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.294026 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.294031 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.294035 | controller |     - '        '
2026-03-14 09:44:27.294039 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.294043 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.294047 | controller |     - '        '
2026-03-14 09:44:27.294052 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.294056 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.294060 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.294064 | controller |     - '          exit 1'
2026-03-14 09:44:27.294068 | controller |     - '        fi'
2026-03-14 09:44:27.294073 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.294077 | controller |     - '        '
2026-03-14 09:44:27.294083 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.294087 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.294091 | controller |       | base64 -d)'
2026-03-14 09:44:27.294103 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.294110 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.294116 | controller |     - '          exit 1'
2026-03-14 09:44:27.294122 | controller |     - '        fi'
2026-03-14 09:44:27.294128 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.294134 | controller |     - '        '
2026-03-14 09:44:27.294139 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.294143 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.294147 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.294151 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.294156 | controller |       --timeout=300s'
2026-03-14 09:44:27.294160 | controller |     - '        '
2026-03-14 09:44:27.294164 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.294168 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.294172 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.294177 | controller |     - '        '
2026-03-14 09:44:27.294181 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.294185 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.294189 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.294193 | controller |     - '        '
2026-03-14 09:44:27.294197 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.294202 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.294206 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.294210 | controller |       available"'
2026-03-14 09:44:27.294214 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.294218 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.294223 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.294227 | controller |     - '          exit 1'
2026-03-14 09:44:27.294231 | controller |     - '        fi'
2026-03-14 09:44:27.294235 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.294242 | controller |     - '        '
2026-03-14 09:44:27.294246 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.294250 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.294254 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.294259 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.294263 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.294267 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.294271 | controller |       -- \'
2026-03-14 09:44:27.294275 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.294282 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.294287 | controller |     - '    logger.go:42: 09:29:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.294307 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.294312 | controller |     - '    logger.go:42: 09:29:31 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.294316 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.294320 | controller |       found'
2026-03-14 09:44:27.294325 | controller |     - '    logger.go:42: 09:29:32 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.294329 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.294333 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.294337 | controller |     - '        '
2026-03-14 09:44:27.294341 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.294346 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.294350 | controller |     - '        '
2026-03-14 09:44:27.294354 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.294358 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.294362 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.294367 | controller |     - '          exit 1'
2026-03-14 09:44:27.294371 | controller |     - '        fi'
2026-03-14 09:44:27.294375 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.294379 | controller |     - '        '
2026-03-14 09:44:27.294383 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.294388 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.294392 | controller |       | base64 -d)'
2026-03-14 09:44:27.294396 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.294407 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.294413 | controller |     - '          exit 1'
2026-03-14 09:44:27.294420 | controller |     - '        fi'
2026-03-14 09:44:27.294426 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.294432 | controller |     - '        '
2026-03-14 09:44:27.294437 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.294442 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.294446 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.294450 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.294454 | controller |       --timeout=300s'
2026-03-14 09:44:27.294458 | controller |     - '        '
2026-03-14 09:44:27.294462 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.294469 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.294473 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.294478 | controller |     - '        '
2026-03-14 09:44:27.294482 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.294486 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.294491 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.294495 | controller |     - '        '
2026-03-14 09:44:27.294499 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.294503 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.294508 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.294512 | controller |       available"'
2026-03-14 09:44:27.294516 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.294520 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.294525 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.294529 | controller |     - '          exit 1'
2026-03-14 09:44:27.294533 | controller |     - '        fi'
2026-03-14 09:44:27.294538 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.294542 | controller |     - '        '
2026-03-14 09:44:27.294546 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.294550 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.294555 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.294559 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.294563 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.294567 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.294571 | controller |       -- \'
2026-03-14 09:44:27.294576 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.294580 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.294587 | controller |     - '    logger.go:42: 09:29:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.294591 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.294595 | controller |     - '    logger.go:42: 09:29:33 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.294600 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.294604 | controller |       found'
2026-03-14 09:44:27.294608 | controller |     - '    logger.go:42: 09:29:34 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.294613 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.294617 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.294621 | controller |     - '        '
2026-03-14 09:44:27.294625 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.294630 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.294634 | controller |     - '        '
2026-03-14 09:44:27.294638 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.294654 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.294659 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.294663 | controller |     - '          exit 1'
2026-03-14 09:44:27.294667 | controller |     - '        fi'
2026-03-14 09:44:27.294672 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.294676 | controller |     - '        '
2026-03-14 09:44:27.294680 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.294684 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.294689 | controller |       | base64 -d)'
2026-03-14 09:44:27.294693 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.294704 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.294711 | controller |     - '          exit 1'
2026-03-14 09:44:27.294717 | controller |     - '        fi'
2026-03-14 09:44:27.294724 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.294730 | controller |     - '        '
2026-03-14 09:44:27.294735 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.294739 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.294743 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.294748 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.294752 | controller |       --timeout=300s'
2026-03-14 09:44:27.294756 | controller |     - '        '
2026-03-14 09:44:27.294761 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.294765 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.294769 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.294773 | controller |     - '        '
2026-03-14 09:44:27.294778 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.294782 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.294786 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.294791 | controller |     - '        '
2026-03-14 09:44:27.294795 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.294802 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.294807 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.294811 | controller |       available"'
2026-03-14 09:44:27.294815 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.294820 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.294824 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.294828 | controller |     - '          exit 1'
2026-03-14 09:44:27.294833 | controller |     - '        fi'
2026-03-14 09:44:27.294837 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.294841 | controller |     - '        '
2026-03-14 09:44:27.294845 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.294850 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.294854 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.294858 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.294864 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.294869 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.294874 | controller |       -- \'
2026-03-14 09:44:27.294878 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.294882 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.294887 | controller |     - '    logger.go:42: 09:29:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.294891 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.294897 | controller |     - '    logger.go:42: 09:29:34 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.294902 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.294906 | controller |       found'
2026-03-14 09:44:27.294910 | controller |     - '    logger.go:42: 09:29:35 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.294914 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.294919 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.294923 | controller |     - '        '
2026-03-14 09:44:27.294927 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.294932 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.294936 | controller |     - '        '
2026-03-14 09:44:27.294940 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.294944 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.294949 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.294953 | controller |     - '          exit 1'
2026-03-14 09:44:27.294957 | controller |     - '        fi'
2026-03-14 09:44:27.294962 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.294966 | controller |     - '        '
2026-03-14 09:44:27.294970 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.294974 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.294979 | controller |       | base64 -d)'
2026-03-14 09:44:27.294983 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.294987 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.294999 | controller |     - '          exit 1'
2026-03-14 09:44:27.295005 | controller |     - '        fi'
2026-03-14 09:44:27.295012 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.295018 | controller |     - '        '
2026-03-14 09:44:27.295027 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.295031 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.295035 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.295040 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.295044 | controller |       --timeout=300s'
2026-03-14 09:44:27.295048 | controller |     - '        '
2026-03-14 09:44:27.295052 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.295057 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.295061 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.295065 | controller |     - '        '
2026-03-14 09:44:27.295069 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.295073 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.295081 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.295085 | controller |     - '        '
2026-03-14 09:44:27.295089 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.295094 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.295098 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.295102 | controller |       available"'
2026-03-14 09:44:27.295106 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.295111 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.295115 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.295119 | controller |     - '          exit 1'
2026-03-14 09:44:27.295123 | controller |     - '        fi'
2026-03-14 09:44:27.295127 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.295132 | controller |     - '        '
2026-03-14 09:44:27.295136 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.295140 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.295144 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.295148 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.295152 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.295157 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.295161 | controller |       -- \'
2026-03-14 09:44:27.295165 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.295169 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.295173 | controller |     - '    logger.go:42: 09:29:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.295178 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.295182 | controller |     - '    logger.go:42: 09:29:35 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.295186 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.295190 | controller |       found'
2026-03-14 09:44:27.295194 | controller |     - '    logger.go:42: 09:29:36 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.295199 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.295203 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.295207 | controller |     - '        '
2026-03-14 09:44:27.295211 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.295215 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.295220 | controller |     - '        '
2026-03-14 09:44:27.295224 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.295228 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.295232 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.295236 | controller |     - '          exit 1'
2026-03-14 09:44:27.295241 | controller |     - '        fi'
2026-03-14 09:44:27.295245 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.295249 | controller |     - '        '
2026-03-14 09:44:27.295253 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.295257 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.295263 | controller |       | base64 -d)'
2026-03-14 09:44:27.295267 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.295272 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.295276 | controller |     - '          exit 1'
2026-03-14 09:44:27.295280 | controller |     - '        fi'
2026-03-14 09:44:27.295306 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.295317 | controller |     - '        '
2026-03-14 09:44:27.295322 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.295327 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.295331 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.295335 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.295339 | controller |       --timeout=300s'
2026-03-14 09:44:27.295344 | controller |     - '        '
2026-03-14 09:44:27.295348 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.295352 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.295357 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.295361 | controller |     - '        '
2026-03-14 09:44:27.295365 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.295370 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.295374 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.295378 | controller |     - '        '
2026-03-14 09:44:27.295382 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.295387 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.295391 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.295395 | controller |       available"'
2026-03-14 09:44:27.295400 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.295407 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.295411 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.295416 | controller |     - '          exit 1'
2026-03-14 09:44:27.295420 | controller |     - '        fi'
2026-03-14 09:44:27.295424 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.295428 | controller |     - '        '
2026-03-14 09:44:27.295433 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.295437 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.295441 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.295445 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.295450 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.295454 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.295458 | controller |       -- \'
2026-03-14 09:44:27.295462 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.295467 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.295471 | controller |     - '    logger.go:42: 09:29:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.295478 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.295482 | controller |     - '    logger.go:42: 09:29:36 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.295499 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.295504 | controller |       found'
2026-03-14 09:44:27.295508 | controller |     - '    logger.go:42: 09:29:37 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.295512 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.295517 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.295521 | controller |     - '        '
2026-03-14 09:44:27.295525 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.295529 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.295534 | controller |     - '        '
2026-03-14 09:44:27.295538 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.295542 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.295546 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.295551 | controller |     - '          exit 1'
2026-03-14 09:44:27.295555 | controller |     - '        fi'
2026-03-14 09:44:27.295559 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.295564 | controller |     - '        '
2026-03-14 09:44:27.295568 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.295572 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.295576 | controller |       | base64 -d)'
2026-03-14 09:44:27.295581 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.295585 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.295589 | controller |     - '          exit 1'
2026-03-14 09:44:27.295593 | controller |     - '        fi'
2026-03-14 09:44:27.295598 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.295610 | controller |     - '        '
2026-03-14 09:44:27.295617 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.295623 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.295629 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.295635 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.295640 | controller |       --timeout=300s'
2026-03-14 09:44:27.295645 | controller |     - '        '
2026-03-14 09:44:27.295649 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.295653 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.295657 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.295662 | controller |     - '        '
2026-03-14 09:44:27.295666 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.295670 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.295674 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.295680 | controller |     - '        '
2026-03-14 09:44:27.295685 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.295691 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.295696 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.295703 | controller |       available"'
2026-03-14 09:44:27.295708 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.295712 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.295716 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.295720 | controller |     - '          exit 1'
2026-03-14 09:44:27.295725 | controller |     - '        fi'
2026-03-14 09:44:27.295729 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.295733 | controller |     - '        '
2026-03-14 09:44:27.295737 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.295741 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.295745 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.295750 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.295754 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.295758 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.295762 | controller |       -- \'
2026-03-14 09:44:27.295766 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.295771 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.295775 | controller |     - '    logger.go:42: 09:29:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.295779 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.295783 | controller |     - '    logger.go:42: 09:29:37 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.295787 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.295792 | controller |       found'
2026-03-14 09:44:27.295796 | controller |     - '    logger.go:42: 09:29:38 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.295800 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.295804 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.295809 | controller |     - '        '
2026-03-14 09:44:27.295814 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.295819 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.295835 | controller |     - '        '
2026-03-14 09:44:27.295844 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.295850 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.295856 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.295863 | controller |     - '          exit 1'
2026-03-14 09:44:27.295869 | controller |     - '        fi'
2026-03-14 09:44:27.295874 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.295880 | controller |     - '        '
2026-03-14 09:44:27.295886 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.295892 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.295897 | controller |       | base64 -d)'
2026-03-14 09:44:27.295903 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.295908 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.295914 | controller |     - '          exit 1'
2026-03-14 09:44:27.295919 | controller |     - '        fi'
2026-03-14 09:44:27.295925 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.295930 | controller |     - '        '
2026-03-14 09:44:27.295946 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.295957 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.295962 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.295968 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.295974 | controller |       --timeout=300s'
2026-03-14 09:44:27.295979 | controller |     - '        '
2026-03-14 09:44:27.295985 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.295991 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.296001 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.296007 | controller |     - '        '
2026-03-14 09:44:27.296015 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.296020 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.296026 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.296033 | controller |     - '        '
2026-03-14 09:44:27.296039 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.296045 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.296050 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.296056 | controller |       available"'
2026-03-14 09:44:27.296062 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.296068 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.296074 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.296080 | controller |     - '          exit 1'
2026-03-14 09:44:27.296086 | controller |     - '        fi'
2026-03-14 09:44:27.296092 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.296098 | controller |     - '        '
2026-03-14 09:44:27.296104 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.296108 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.296112 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.296116 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.296121 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.296125 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.296129 | controller |       -- \'
2026-03-14 09:44:27.296133 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.296137 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.296142 | controller |     - '    logger.go:42: 09:29:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.296146 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.296150 | controller |     - '    logger.go:42: 09:29:38 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.296154 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.296158 | controller |       found'
2026-03-14 09:44:27.296166 | controller |     - '    logger.go:42: 09:29:39 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.296170 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.296176 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.296180 | controller |     - '        '
2026-03-14 09:44:27.296185 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.296189 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.296193 | controller |     - '        '
2026-03-14 09:44:27.296197 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.296201 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.296207 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.296212 | controller |     - '          exit 1'
2026-03-14 09:44:27.296216 | controller |     - '        fi'
2026-03-14 09:44:27.296222 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.296226 | controller |     - '        '
2026-03-14 09:44:27.296230 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.296235 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.296239 | controller |       | base64 -d)'
2026-03-14 09:44:27.296243 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.296247 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.296252 | controller |     - '          exit 1'
2026-03-14 09:44:27.296258 | controller |     - '        fi'
2026-03-14 09:44:27.296264 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.296270 | controller |     - '        '
2026-03-14 09:44:27.296276 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.296306 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.296317 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.296322 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.296326 | controller |       --timeout=300s'
2026-03-14 09:44:27.296331 | controller |     - '        '
2026-03-14 09:44:27.296336 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.296352 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.296361 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.296367 | controller |     - '        '
2026-03-14 09:44:27.296373 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.296378 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.296384 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.296390 | controller |     - '        '
2026-03-14 09:44:27.296396 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.296402 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.296407 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.296412 | controller |       available"'
2026-03-14 09:44:27.296416 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.296420 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.296424 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.296429 | controller |     - '          exit 1'
2026-03-14 09:44:27.296433 | controller |     - '        fi'
2026-03-14 09:44:27.296437 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.296445 | controller |     - '        '
2026-03-14 09:44:27.296449 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.296453 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.296457 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.296461 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.296466 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.296470 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.296474 | controller |       -- \'
2026-03-14 09:44:27.296478 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.296482 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.296487 | controller |     - '    logger.go:42: 09:29:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.296491 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.296495 | controller |     - '    logger.go:42: 09:29:39 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.296499 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.296504 | controller |       found'
2026-03-14 09:44:27.296508 | controller |     - '    logger.go:42: 09:29:40 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.296512 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.296517 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.296521 | controller |     - '        '
2026-03-14 09:44:27.296525 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.296529 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.296533 | controller |     - '        '
2026-03-14 09:44:27.296537 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.296542 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.296546 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.296552 | controller |     - '          exit 1'
2026-03-14 09:44:27.296557 | controller |     - '        fi'
2026-03-14 09:44:27.296563 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.296568 | controller |     - '        '
2026-03-14 09:44:27.296573 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.296579 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.296585 | controller |       | base64 -d)'
2026-03-14 09:44:27.296590 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.296596 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.296601 | controller |     - '          exit 1'
2026-03-14 09:44:27.296607 | controller |     - '        fi'
2026-03-14 09:44:27.296612 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.296618 | controller |     - '        '
2026-03-14 09:44:27.296623 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.296629 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.296645 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.296651 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.296657 | controller |       --timeout=300s'
2026-03-14 09:44:27.296662 | controller |     - '        '
2026-03-14 09:44:27.296667 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.296675 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.296681 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.296687 | controller |     - '        '
2026-03-14 09:44:27.296693 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.296699 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.296705 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.296710 | controller |     - '        '
2026-03-14 09:44:27.296716 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.296722 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.296727 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.296733 | controller |       available"'
2026-03-14 09:44:27.296738 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.296744 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.296749 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.296755 | controller |     - '          exit 1'
2026-03-14 09:44:27.296761 | controller |     - '        fi'
2026-03-14 09:44:27.296766 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.296771 | controller |     - '        '
2026-03-14 09:44:27.296777 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.296781 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.296785 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.296791 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.296803 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.296814 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.296828 | controller |       -- \'
2026-03-14 09:44:27.296836 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.296841 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.296847 | controller |     - '    logger.go:42: 09:29:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.296854 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.296860 | controller |     - '    logger.go:42: 09:29:41 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.296866 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.296872 | controller |       found'
2026-03-14 09:44:27.296878 | controller |     - '    logger.go:42: 09:29:42 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.296884 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.296890 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.296896 | controller |     - '        '
2026-03-14 09:44:27.296905 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.296911 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.296917 | controller |     - '        '
2026-03-14 09:44:27.296923 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.296929 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.296938 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.296944 | controller |     - '          exit 1'
2026-03-14 09:44:27.296950 | controller |     - '        fi'
2026-03-14 09:44:27.296956 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.296962 | controller |     - '        '
2026-03-14 09:44:27.296971 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.296977 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.296983 | controller |       | base64 -d)'
2026-03-14 09:44:27.296989 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.296995 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.297000 | controller |     - '          exit 1'
2026-03-14 09:44:27.297006 | controller |     - '        fi'
2026-03-14 09:44:27.297012 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.297018 | controller |     - '        '
2026-03-14 09:44:27.297024 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.297029 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.297045 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.297052 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.297058 | controller |       --timeout=300s'
2026-03-14 09:44:27.297063 | controller |     - '        '
2026-03-14 09:44:27.297069 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.297075 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.297081 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.297087 | controller |     - '        '
2026-03-14 09:44:27.297092 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.297098 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.297104 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.297110 | controller |     - '        '
2026-03-14 09:44:27.297115 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.297121 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.297126 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.297132 | controller |       available"'
2026-03-14 09:44:27.297138 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.297143 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.297149 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.297155 | controller |     - '          exit 1'
2026-03-14 09:44:27.297160 | controller |     - '        fi'
2026-03-14 09:44:27.297166 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.297172 | controller |     - '        '
2026-03-14 09:44:27.297177 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.297183 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.297188 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.297194 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.297203 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.297211 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.297216 | controller |       -- \'
2026-03-14 09:44:27.297222 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.297228 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.297233 | controller |     - '    logger.go:42: 09:29:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.297239 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.297244 | controller |     - '    logger.go:42: 09:29:42 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.297250 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.297255 | controller |       found'
2026-03-14 09:44:27.297261 | controller |     - '    logger.go:42: 09:29:43 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.297267 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.297273 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.297278 | controller |     - '        '
2026-03-14 09:44:27.297284 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.297310 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.297316 | controller |     - '        '
2026-03-14 09:44:27.297321 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.297327 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.297332 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.297338 | controller |     - '          exit 1'
2026-03-14 09:44:27.297343 | controller |     - '        fi'
2026-03-14 09:44:27.297349 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.297355 | controller |     - '        '
2026-03-14 09:44:27.297360 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.297366 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.297371 | controller |       | base64 -d)'
2026-03-14 09:44:27.297377 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.297383 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.297388 | controller |     - '          exit 1'
2026-03-14 09:44:27.297394 | controller |     - '        fi'
2026-03-14 09:44:27.297399 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.297405 | controller |     - '        '
2026-03-14 09:44:27.297410 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.297416 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.297421 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.297435 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.297441 | controller |       --timeout=300s'
2026-03-14 09:44:27.297446 | controller |     - '        '
2026-03-14 09:44:27.297452 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.297457 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.297463 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.297468 | controller |     - '        '
2026-03-14 09:44:27.297473 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.297479 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.297487 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.297493 | controller |     - '        '
2026-03-14 09:44:27.297498 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.297504 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.297509 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.297515 | controller |       available"'
2026-03-14 09:44:27.297521 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.297526 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.297531 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.297537 | controller |     - '          exit 1'
2026-03-14 09:44:27.297542 | controller |     - '        fi'
2026-03-14 09:44:27.297551 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.297556 | controller |     - '        '
2026-03-14 09:44:27.297562 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.297567 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.297572 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.297578 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.297583 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.297589 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.297594 | controller |       -- \'
2026-03-14 09:44:27.297599 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.297604 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.297610 | controller |     - '    logger.go:42: 09:29:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.297616 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.297621 | controller |     - '    logger.go:42: 09:29:43 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.297627 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.297632 | controller |       found'
2026-03-14 09:44:27.297638 | controller |     - '    logger.go:42: 09:29:44 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.297643 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.297649 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.297654 | controller |     - '        '
2026-03-14 09:44:27.297660 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.297667 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.297673 | controller |     - '        '
2026-03-14 09:44:27.297678 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.297684 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.297690 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.297695 | controller |     - '          exit 1'
2026-03-14 09:44:27.297701 | controller |     - '        fi'
2026-03-14 09:44:27.297706 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.297712 | controller |     - '        '
2026-03-14 09:44:27.297718 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.297723 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.297729 | controller |       | base64 -d)'
2026-03-14 09:44:27.297737 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.297745 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.297751 | controller |     - '          exit 1'
2026-03-14 09:44:27.297756 | controller |     - '        fi'
2026-03-14 09:44:27.297762 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.297767 | controller |     - '        '
2026-03-14 09:44:27.297773 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.297779 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.297784 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.297798 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.297805 | controller |       --timeout=300s'
2026-03-14 09:44:27.297809 | controller |     - '        '
2026-03-14 09:44:27.297814 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.297818 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.297822 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.297826 | controller |     - '        '
2026-03-14 09:44:27.297830 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.297835 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.297839 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.297843 | controller |     - '        '
2026-03-14 09:44:27.297847 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.297851 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.297856 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.297860 | controller |       available"'
2026-03-14 09:44:27.297864 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.297868 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.297872 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.297877 | controller |     - '          exit 1'
2026-03-14 09:44:27.297881 | controller |     - '        fi'
2026-03-14 09:44:27.297885 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.297889 | controller |     - '        '
2026-03-14 09:44:27.297893 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.297912 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.297918 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.297922 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.297927 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.297931 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.297935 | controller |       -- \'
2026-03-14 09:44:27.297940 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.297944 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.297948 | controller |     - '    logger.go:42: 09:29:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.297955 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.297960 | controller |     - '    logger.go:42: 09:29:44 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.297964 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.297969 | controller |       found'
2026-03-14 09:44:27.297973 | controller |     - '    logger.go:42: 09:29:45 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.297977 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.297981 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.297986 | controller |     - '        '
2026-03-14 09:44:27.297990 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.297994 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.297998 | controller |     - '        '
2026-03-14 09:44:27.298003 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.298007 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.298011 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.298015 | controller |     - '          exit 1'
2026-03-14 09:44:27.298020 | controller |     - '        fi'
2026-03-14 09:44:27.298024 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.298028 | controller |     - '        '
2026-03-14 09:44:27.298032 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.298037 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.298041 | controller |       | base64 -d)'
2026-03-14 09:44:27.298045 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.298049 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.298054 | controller |     - '          exit 1'
2026-03-14 09:44:27.298058 | controller |     - '        fi'
2026-03-14 09:44:27.298062 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.298066 | controller |     - '        '
2026-03-14 09:44:27.298071 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.298075 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.298079 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.298083 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.298088 | controller |       --timeout=300s'
2026-03-14 09:44:27.298101 | controller |     - '        '
2026-03-14 09:44:27.298106 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.298110 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.298114 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.298118 | controller |     - '        '
2026-03-14 09:44:27.298123 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.298127 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.298131 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.298138 | controller |     - '        '
2026-03-14 09:44:27.298142 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.298147 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.298151 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.298155 | controller |       available"'
2026-03-14 09:44:27.298161 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.298165 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.298170 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.298174 | controller |     - '          exit 1'
2026-03-14 09:44:27.298178 | controller |     - '        fi'
2026-03-14 09:44:27.298183 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.298187 | controller |     - '        '
2026-03-14 09:44:27.298191 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.298195 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.298200 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.298204 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.298208 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.298213 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.298217 | controller |       -- \'
2026-03-14 09:44:27.298221 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.298225 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.298230 | controller |     - '    logger.go:42: 09:29:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.298234 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.298238 | controller |     - '    logger.go:42: 09:29:45 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.298242 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.298247 | controller |       found'
2026-03-14 09:44:27.298251 | controller |     - '    logger.go:42: 09:29:46 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.298257 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.298263 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.298270 | controller |     - '        '
2026-03-14 09:44:27.298276 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.298281 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.298303 | controller |     - '        '
2026-03-14 09:44:27.298312 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.298317 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.298321 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.298326 | controller |     - '          exit 1'
2026-03-14 09:44:27.298334 | controller |     - '        fi'
2026-03-14 09:44:27.298338 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.298343 | controller |     - '        '
2026-03-14 09:44:27.298347 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.298351 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.298356 | controller |       | base64 -d)'
2026-03-14 09:44:27.298360 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.298364 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.298368 | controller |     - '          exit 1'
2026-03-14 09:44:27.298373 | controller |     - '        fi'
2026-03-14 09:44:27.298377 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.298381 | controller |     - '        '
2026-03-14 09:44:27.298385 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.298389 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.298396 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.298400 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.298404 | controller |       --timeout=300s'
2026-03-14 09:44:27.298408 | controller |     - '        '
2026-03-14 09:44:27.298420 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.298425 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.298429 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.298433 | controller |     - '        '
2026-03-14 09:44:27.298438 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.298443 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.298448 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.298452 | controller |     - '        '
2026-03-14 09:44:27.298456 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.298461 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.298465 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.298469 | controller |       available"'
2026-03-14 09:44:27.298473 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.298479 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.298483 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.298488 | controller |     - '          exit 1'
2026-03-14 09:44:27.298492 | controller |     - '        fi'
2026-03-14 09:44:27.298496 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.298500 | controller |     - '        '
2026-03-14 09:44:27.298505 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.298509 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.298513 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.298517 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.298522 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.298526 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.298530 | controller |       -- \'
2026-03-14 09:44:27.298534 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.298539 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.298543 | controller |     - '    logger.go:42: 09:29:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.298547 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.298551 | controller |     - '    logger.go:42: 09:29:46 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.298556 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.298560 | controller |       found'
2026-03-14 09:44:27.298564 | controller |     - '    logger.go:42: 09:29:47 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.298568 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.298573 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.298579 | controller |     - '        '
2026-03-14 09:44:27.298583 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.298587 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.298591 | controller |     - '        '
2026-03-14 09:44:27.298596 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.298600 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.298604 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.298609 | controller |     - '          exit 1'
2026-03-14 09:44:27.298613 | controller |     - '        fi'
2026-03-14 09:44:27.298617 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.298621 | controller |     - '        '
2026-03-14 09:44:27.298626 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.298630 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.298634 | controller |       | base64 -d)'
2026-03-14 09:44:27.298638 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.298643 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.298647 | controller |     - '          exit 1'
2026-03-14 09:44:27.298651 | controller |     - '        fi'
2026-03-14 09:44:27.298655 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.298660 | controller |     - '        '
2026-03-14 09:44:27.298664 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.298668 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.298672 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.298677 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.298681 | controller |       --timeout=300s'
2026-03-14 09:44:27.298685 | controller |     - '        '
2026-03-14 09:44:27.298697 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.298702 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.298706 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.298711 | controller |     - '        '
2026-03-14 09:44:27.298715 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.298719 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.298723 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.298728 | controller |     - '        '
2026-03-14 09:44:27.298732 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.298736 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.298740 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.298745 | controller |       available"'
2026-03-14 09:44:27.298749 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.298753 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.298757 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.298761 | controller |     - '          exit 1'
2026-03-14 09:44:27.298766 | controller |     - '        fi'
2026-03-14 09:44:27.298770 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.298774 | controller |     - '        '
2026-03-14 09:44:27.298778 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.298784 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.298789 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.298793 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.298797 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.298801 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.298805 | controller |       -- \'
2026-03-14 09:44:27.298810 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.298814 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.298818 | controller |     - '    logger.go:42: 09:29:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.298822 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.298827 | controller |     - '    logger.go:42: 09:29:47 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.298831 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.298835 | controller |       found'
2026-03-14 09:44:27.298839 | controller |     - '    logger.go:42: 09:29:48 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.298844 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.298848 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.298852 | controller |     - '        '
2026-03-14 09:44:27.298856 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.298861 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.298865 | controller |     - '        '
2026-03-14 09:44:27.298869 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.298873 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.298877 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.298882 | controller |     - '          exit 1'
2026-03-14 09:44:27.298886 | controller |     - '        fi'
2026-03-14 09:44:27.298890 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.298896 | controller |     - '        '
2026-03-14 09:44:27.298901 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.298905 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.298909 | controller |       | base64 -d)'
2026-03-14 09:44:27.298913 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.298917 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.298922 | controller |     - '          exit 1'
2026-03-14 09:44:27.298926 | controller |     - '        fi'
2026-03-14 09:44:27.298930 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.298934 | controller |     - '        '
2026-03-14 09:44:27.298939 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.298943 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.298947 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.298951 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.298955 | controller |       --timeout=300s'
2026-03-14 09:44:27.298960 | controller |     - '        '
2026-03-14 09:44:27.298964 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.298974 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.298980 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.298985 | controller |     - '        '
2026-03-14 09:44:27.298989 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.298993 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.298997 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.299002 | controller |     - '        '
2026-03-14 09:44:27.299006 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.299010 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.299014 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.299018 | controller |       available"'
2026-03-14 09:44:27.299023 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.299027 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.299031 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.299035 | controller |     - '          exit 1'
2026-03-14 09:44:27.299040 | controller |     - '        fi'
2026-03-14 09:44:27.299044 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.299048 | controller |     - '        '
2026-03-14 09:44:27.299052 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.299057 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.299061 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.299065 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.299069 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.299074 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.299078 | controller |       -- \'
2026-03-14 09:44:27.299082 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.299086 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.299092 | controller |     - '    logger.go:42: 09:29:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.299096 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.299101 | controller |     - '    logger.go:42: 09:29:49 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.299105 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.299109 | controller |       found'
2026-03-14 09:44:27.299113 | controller |     - '    logger.go:42: 09:29:50 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.299117 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.299122 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.299126 | controller |     - '        '
2026-03-14 09:44:27.299130 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.299134 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.299138 | controller |     - '        '
2026-03-14 09:44:27.299143 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.299147 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.299151 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.299157 | controller |     - '          exit 1'
2026-03-14 09:44:27.299161 | controller |     - '        fi'
2026-03-14 09:44:27.299166 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.299170 | controller |     - '        '
2026-03-14 09:44:27.299174 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.299178 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.299183 | controller |       | base64 -d)'
2026-03-14 09:44:27.299187 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.299191 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.299195 | controller |     - '          exit 1'
2026-03-14 09:44:27.299199 | controller |     - '        fi'
2026-03-14 09:44:27.299204 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.299208 | controller |     - '        '
2026-03-14 09:44:27.299212 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.299216 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.299221 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.299225 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.299229 | controller |       --timeout=300s'
2026-03-14 09:44:27.299233 | controller |     - '        '
2026-03-14 09:44:27.299237 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.299248 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.299253 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.299259 | controller |     - '        '
2026-03-14 09:44:27.299265 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.299271 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.299277 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.299283 | controller |     - '        '
2026-03-14 09:44:27.299304 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.299312 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.299316 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.299321 | controller |       available"'
2026-03-14 09:44:27.299325 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.299329 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.299333 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.299337 | controller |     - '          exit 1'
2026-03-14 09:44:27.299342 | controller |     - '        fi'
2026-03-14 09:44:27.299346 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.299350 | controller |     - '        '
2026-03-14 09:44:27.299354 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.299358 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.299363 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.299367 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.299371 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.299379 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.299385 | controller |       -- \'
2026-03-14 09:44:27.299390 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.299394 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.299398 | controller |     - '    logger.go:42: 09:29:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.299402 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.299406 | controller |     - '    logger.go:42: 09:29:50 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.299411 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.299415 | controller |       found'
2026-03-14 09:44:27.299419 | controller |     - '    logger.go:42: 09:29:51 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.299423 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.299427 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.299431 | controller |     - '        '
2026-03-14 09:44:27.299436 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.299440 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.299444 | controller |     - '        '
2026-03-14 09:44:27.299448 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.299452 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.299456 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.299461 | controller |     - '          exit 1'
2026-03-14 09:44:27.299465 | controller |     - '        fi'
2026-03-14 09:44:27.299469 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.299473 | controller |     - '        '
2026-03-14 09:44:27.299477 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.299482 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.299486 | controller |       | base64 -d)'
2026-03-14 09:44:27.299490 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.299494 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.299498 | controller |     - '          exit 1'
2026-03-14 09:44:27.299503 | controller |     - '        fi'
2026-03-14 09:44:27.299507 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.299511 | controller |     - '        '
2026-03-14 09:44:27.299515 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.299519 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.299525 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.299529 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.299533 | controller |       --timeout=300s'
2026-03-14 09:44:27.299538 | controller |     - '        '
2026-03-14 09:44:27.299542 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.299546 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.299557 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.299562 | controller |     - '        '
2026-03-14 09:44:27.299566 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.299570 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.299575 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.299581 | controller |     - '        '
2026-03-14 09:44:27.299585 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.299589 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.299593 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.299597 | controller |       available"'
2026-03-14 09:44:27.299602 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.299606 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.299610 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.299614 | controller |     - '          exit 1'
2026-03-14 09:44:27.299618 | controller |     - '        fi'
2026-03-14 09:44:27.299622 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.299626 | controller |     - '        '
2026-03-14 09:44:27.299631 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.299635 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.299639 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.299643 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.299647 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.299652 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.299656 | controller |       -- \'
2026-03-14 09:44:27.299660 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.299664 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.299668 | controller |     - '    logger.go:42: 09:29:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.299673 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.299677 | controller |     - '    logger.go:42: 09:29:51 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.299681 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.299685 | controller |       found'
2026-03-14 09:44:27.299693 | controller |     - '    logger.go:42: 09:29:52 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.299697 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.299702 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.299706 | controller |     - '        '
2026-03-14 09:44:27.299710 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.299714 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.299718 | controller |     - '        '
2026-03-14 09:44:27.299722 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.299727 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.299731 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.299735 | controller |     - '          exit 1'
2026-03-14 09:44:27.299739 | controller |     - '        fi'
2026-03-14 09:44:27.299743 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.299748 | controller |     - '        '
2026-03-14 09:44:27.299755 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.299759 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.299764 | controller |       | base64 -d)'
2026-03-14 09:44:27.299781 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.299787 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.299792 | controller |     - '          exit 1'
2026-03-14 09:44:27.299796 | controller |     - '        fi'
2026-03-14 09:44:27.299800 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.299804 | controller |     - '        '
2026-03-14 09:44:27.299809 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.299813 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.299817 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.299821 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.299825 | controller |       --timeout=300s'
2026-03-14 09:44:27.299830 | controller |     - '        '
2026-03-14 09:44:27.299834 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.299838 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.299849 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.299854 | controller |     - '        '
2026-03-14 09:44:27.299858 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.299862 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.299866 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.299871 | controller |     - '        '
2026-03-14 09:44:27.299875 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.299879 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.299883 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.299887 | controller |       available"'
2026-03-14 09:44:27.299892 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.299896 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.299900 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.299904 | controller |     - '          exit 1'
2026-03-14 09:44:27.299908 | controller |     - '        fi'
2026-03-14 09:44:27.299913 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.299917 | controller |     - '        '
2026-03-14 09:44:27.299921 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.299925 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.299929 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.299934 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.299938 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.299942 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.299946 | controller |       -- \'
2026-03-14 09:44:27.299950 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.299955 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.299959 | controller |     - '    logger.go:42: 09:29:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.299963 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.299969 | controller |     - '    logger.go:42: 09:29:52 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.299974 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.299978 | controller |       found'
2026-03-14 09:44:27.299982 | controller |     - '    logger.go:42: 09:29:53 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.299986 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.299990 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.299995 | controller |     - '        '
2026-03-14 09:44:27.299999 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.300003 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.300007 | controller |     - '        '
2026-03-14 09:44:27.300011 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.300015 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.300020 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.300024 | controller |     - '          exit 1'
2026-03-14 09:44:27.300028 | controller |     - '        fi'
2026-03-14 09:44:27.300032 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.300036 | controller |     - '        '
2026-03-14 09:44:27.300041 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.300045 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.300049 | controller |       | base64 -d)'
2026-03-14 09:44:27.300053 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.300057 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.300062 | controller |     - '          exit 1'
2026-03-14 09:44:27.300066 | controller |     - '        fi'
2026-03-14 09:44:27.300070 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.300074 | controller |     - '        '
2026-03-14 09:44:27.300078 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.300082 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.300087 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.300093 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.300098 | controller |       --timeout=300s'
2026-03-14 09:44:27.300102 | controller |     - '        '
2026-03-14 09:44:27.300106 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.300110 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.300115 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.300119 | controller |     - '        '
2026-03-14 09:44:27.300129 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.300133 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.300137 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.300142 | controller |     - '        '
2026-03-14 09:44:27.300146 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.300150 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.300154 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.300159 | controller |       available"'
2026-03-14 09:44:27.300163 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.300169 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.300173 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.300178 | controller |     - '          exit 1'
2026-03-14 09:44:27.300182 | controller |     - '        fi'
2026-03-14 09:44:27.300186 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.300190 | controller |     - '        '
2026-03-14 09:44:27.300195 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.300199 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.300203 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.300207 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.300212 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.300216 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.300220 | controller |       -- \'
2026-03-14 09:44:27.300224 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.300228 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.300233 | controller |     - '    logger.go:42: 09:29:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.300237 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.300241 | controller |     - '    logger.go:42: 09:29:53 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.300245 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.300250 | controller |       found'
2026-03-14 09:44:27.300256 | controller |     - '    logger.go:42: 09:29:54 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.300261 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.300271 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.300277 | controller |     - '        '
2026-03-14 09:44:27.300283 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.300303 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.300313 | controller |     - '        '
2026-03-14 09:44:27.300318 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.300324 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.300330 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.300336 | controller |     - '          exit 1'
2026-03-14 09:44:27.300342 | controller |     - '        fi'
2026-03-14 09:44:27.300347 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.300353 | controller |     - '        '
2026-03-14 09:44:27.300358 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.300364 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.300369 | controller |       | base64 -d)'
2026-03-14 09:44:27.300374 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.300379 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.300383 | controller |     - '          exit 1'
2026-03-14 09:44:27.300387 | controller |     - '        fi'
2026-03-14 09:44:27.300391 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.300395 | controller |     - '        '
2026-03-14 09:44:27.300400 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.300404 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.300411 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.300415 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.300420 | controller |       --timeout=300s'
2026-03-14 09:44:27.300424 | controller |     - '        '
2026-03-14 09:44:27.300428 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.300432 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.300436 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.300441 | controller |     - '        '
2026-03-14 09:44:27.300462 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.300468 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.300474 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.300478 | controller |     - '        '
2026-03-14 09:44:27.300483 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.300487 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.300491 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.300495 | controller |       available"'
2026-03-14 09:44:27.300499 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.300504 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.300508 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.300512 | controller |     - '          exit 1'
2026-03-14 09:44:27.300516 | controller |     - '        fi'
2026-03-14 09:44:27.300520 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.300524 | controller |     - '        '
2026-03-14 09:44:27.300528 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.300533 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.300537 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.300541 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.300547 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.300551 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.300555 | controller |       -- \'
2026-03-14 09:44:27.300559 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.300564 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.300568 | controller |     - '    logger.go:42: 09:29:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.300572 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.300576 | controller |     - '    logger.go:42: 09:29:54 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.300580 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.300584 | controller |       found'
2026-03-14 09:44:27.300589 | controller |     - '    logger.go:42: 09:29:55 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.300593 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.300597 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.300601 | controller |     - '        '
2026-03-14 09:44:27.300605 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.300612 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.300616 | controller |     - '        '
2026-03-14 09:44:27.300620 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.300624 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.300628 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.300632 | controller |     - '          exit 1'
2026-03-14 09:44:27.300636 | controller |     - '        fi'
2026-03-14 09:44:27.300641 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.300645 | controller |     - '        '
2026-03-14 09:44:27.300649 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.300653 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.300657 | controller |       | base64 -d)'
2026-03-14 09:44:27.300661 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.300666 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.300670 | controller |     - '          exit 1'
2026-03-14 09:44:27.300674 | controller |     - '        fi'
2026-03-14 09:44:27.300678 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.300682 | controller |     - '        '
2026-03-14 09:44:27.300686 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.300691 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.300695 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.300699 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.300703 | controller |       --timeout=300s'
2026-03-14 09:44:27.300707 | controller |     - '        '
2026-03-14 09:44:27.300711 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.300716 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.300720 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.300724 | controller |     - '        '
2026-03-14 09:44:27.300728 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.300741 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.300745 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.300749 | controller |     - '        '
2026-03-14 09:44:27.300754 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.300758 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.300762 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.300766 | controller |       available"'
2026-03-14 09:44:27.300770 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.300775 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.300779 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.300783 | controller |     - '          exit 1'
2026-03-14 09:44:27.300787 | controller |     - '        fi'
2026-03-14 09:44:27.300791 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.300795 | controller |     - '        '
2026-03-14 09:44:27.300800 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.300812 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.300817 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.300821 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.300825 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.300829 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.300834 | controller |       -- \'
2026-03-14 09:44:27.300838 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.300842 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.300846 | controller |     - '    logger.go:42: 09:29:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.300850 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.300855 | controller |     - '    logger.go:42: 09:29:55 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.300859 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.300863 | controller |       found'
2026-03-14 09:44:27.300867 | controller |     - '    logger.go:42: 09:29:56 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.300872 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.300877 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.300883 | controller |     - '        '
2026-03-14 09:44:27.300891 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.300897 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.300901 | controller |     - '        '
2026-03-14 09:44:27.300905 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.300910 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.300914 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.300918 | controller |     - '          exit 1'
2026-03-14 09:44:27.300922 | controller |     - '        fi'
2026-03-14 09:44:27.300926 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.300931 | controller |     - '        '
2026-03-14 09:44:27.300935 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.300939 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.300943 | controller |       | base64 -d)'
2026-03-14 09:44:27.300948 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.300952 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.300956 | controller |     - '          exit 1'
2026-03-14 09:44:27.300960 | controller |     - '        fi'
2026-03-14 09:44:27.300964 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.300969 | controller |     - '        '
2026-03-14 09:44:27.300973 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.300977 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.300981 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.300985 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.300989 | controller |       --timeout=300s'
2026-03-14 09:44:27.300994 | controller |     - '        '
2026-03-14 09:44:27.300998 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.301002 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.301013 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.301017 | controller |     - '        '
2026-03-14 09:44:27.301021 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.301032 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.301037 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.301041 | controller |     - '        '
2026-03-14 09:44:27.301045 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.301050 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.301054 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.301058 | controller |       available"'
2026-03-14 09:44:27.301062 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.301069 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.301073 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.301078 | controller |     - '          exit 1'
2026-03-14 09:44:27.301082 | controller |     - '        fi'
2026-03-14 09:44:27.301086 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.301090 | controller |     - '        '
2026-03-14 09:44:27.301095 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.301099 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.301103 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.301107 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.301111 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.301116 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.301120 | controller |       -- \'
2026-03-14 09:44:27.301124 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.301129 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.301133 | controller |     - '    logger.go:42: 09:29:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.301137 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.301141 | controller |     - '    logger.go:42: 09:29:57 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.301146 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.301150 | controller |       found'
2026-03-14 09:44:27.301154 | controller |     - '    logger.go:42: 09:29:58 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.301158 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.301163 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.301167 | controller |     - '        '
2026-03-14 09:44:27.301171 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.301175 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.301179 | controller |     - '        '
2026-03-14 09:44:27.301184 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.301188 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.301192 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.301196 | controller |     - '          exit 1'
2026-03-14 09:44:27.301202 | controller |     - '        fi'
2026-03-14 09:44:27.301207 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.301211 | controller |     - '        '
2026-03-14 09:44:27.301215 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.301219 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.301224 | controller |       | base64 -d)'
2026-03-14 09:44:27.301228 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.301232 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.301236 | controller |     - '          exit 1'
2026-03-14 09:44:27.301240 | controller |     - '        fi'
2026-03-14 09:44:27.301245 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.301249 | controller |     - '        '
2026-03-14 09:44:27.301254 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.301260 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.301266 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.301272 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.301278 | controller |       --timeout=300s'
2026-03-14 09:44:27.301284 | controller |     - '        '
2026-03-14 09:44:27.301306 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.301312 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.301333 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.301337 | controller |     - '        '
2026-03-14 09:44:27.301341 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.301353 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.301357 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.301362 | controller |     - '        '
2026-03-14 09:44:27.301366 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.301370 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.301377 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.301381 | controller |       available"'
2026-03-14 09:44:27.301386 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.301390 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.301394 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.301398 | controller |     - '          exit 1'
2026-03-14 09:44:27.301402 | controller |     - '        fi'
2026-03-14 09:44:27.301407 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.301411 | controller |     - '        '
2026-03-14 09:44:27.301415 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.301419 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.301423 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.301427 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.301432 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.301436 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.301442 | controller |       -- \'
2026-03-14 09:44:27.301447 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.301451 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.301457 | controller |     - '    logger.go:42: 09:29:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.301461 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.301465 | controller |     - '    logger.go:42: 09:29:58 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.301470 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.301474 | controller |       found'
2026-03-14 09:44:27.301478 | controller |     - '    logger.go:42: 09:29:59 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.301482 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.301486 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.301491 | controller |     - '        '
2026-03-14 09:44:27.301495 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.301499 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.301503 | controller |     - '        '
2026-03-14 09:44:27.301507 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.301512 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.301516 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.301520 | controller |     - '          exit 1'
2026-03-14 09:44:27.301524 | controller |     - '        fi'
2026-03-14 09:44:27.301528 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.301533 | controller |     - '        '
2026-03-14 09:44:27.301537 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.301541 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.301545 | controller |       | base64 -d)'
2026-03-14 09:44:27.301549 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.301554 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.301558 | controller |     - '          exit 1'
2026-03-14 09:44:27.301562 | controller |     - '        fi'
2026-03-14 09:44:27.301566 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.301570 | controller |     - '        '
2026-03-14 09:44:27.301575 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.301579 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.301583 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.301587 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.301591 | controller |       --timeout=300s'
2026-03-14 09:44:27.301595 | controller |     - '        '
2026-03-14 09:44:27.301600 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.301604 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.301608 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.301612 | controller |     - '        '
2026-03-14 09:44:27.301616 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.301621 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.301631 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.301644 | controller |     - '        '
2026-03-14 09:44:27.301648 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.301653 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.301657 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.301661 | controller |       available"'
2026-03-14 09:44:27.301665 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.301670 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.301674 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.301678 | controller |     - '          exit 1'
2026-03-14 09:44:27.301682 | controller |     - '        fi'
2026-03-14 09:44:27.301687 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.301691 | controller |     - '        '
2026-03-14 09:44:27.301695 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.301699 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.301703 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.301708 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.301712 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.301716 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.301720 | controller |       -- \'
2026-03-14 09:44:27.301724 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.301728 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.301733 | controller |     - '    logger.go:42: 09:29:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.301737 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.301741 | controller |     - '    logger.go:42: 09:29:59 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.301745 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.301750 | controller |       found'
2026-03-14 09:44:27.301754 | controller |     - '    logger.go:42: 09:30:00 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.301758 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.301762 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.301766 | controller |     - '        '
2026-03-14 09:44:27.301771 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.301775 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.301779 | controller |     - '        '
2026-03-14 09:44:27.301783 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.301788 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.301792 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.301796 | controller |     - '          exit 1'
2026-03-14 09:44:27.301800 | controller |     - '        fi'
2026-03-14 09:44:27.301804 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.301809 | controller |     - '        '
2026-03-14 09:44:27.301813 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.301817 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.301821 | controller |       | base64 -d)'
2026-03-14 09:44:27.301826 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.301832 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.301838 | controller |     - '          exit 1'
2026-03-14 09:44:27.301843 | controller |     - '        fi'
2026-03-14 09:44:27.301847 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.301851 | controller |     - '        '
2026-03-14 09:44:27.301855 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.301860 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.301864 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.301868 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.301872 | controller |       --timeout=300s'
2026-03-14 09:44:27.301876 | controller |     - '        '
2026-03-14 09:44:27.301881 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.301885 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.301889 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.301893 | controller |     - '        '
2026-03-14 09:44:27.301898 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.301902 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.301951 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.301957 | controller |     - '        '
2026-03-14 09:44:27.301964 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.301968 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.301972 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.301977 | controller |       available"'
2026-03-14 09:44:27.301981 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.301987 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.301991 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.301995 | controller |     - '          exit 1'
2026-03-14 09:44:27.301999 | controller |     - '        fi'
2026-03-14 09:44:27.302003 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.302008 | controller |     - '        '
2026-03-14 09:44:27.302012 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.302016 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.302020 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.302024 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.302028 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.302033 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.302037 | controller |       -- \'
2026-03-14 09:44:27.302041 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.302045 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.302049 | controller |     - '    logger.go:42: 09:30:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.302054 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.302058 | controller |     - '    logger.go:42: 09:30:00 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.302064 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.302068 | controller |       found'
2026-03-14 09:44:27.302072 | controller |     - '    logger.go:42: 09:30:01 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.302076 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.302081 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.302085 | controller |     - '        '
2026-03-14 09:44:27.302089 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.302093 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.302097 | controller |     - '        '
2026-03-14 09:44:27.302101 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.302106 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.302110 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.302114 | controller |     - '          exit 1'
2026-03-14 09:44:27.302118 | controller |     - '        fi'
2026-03-14 09:44:27.302122 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.302127 | controller |     - '        '
2026-03-14 09:44:27.302131 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.302135 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.302139 | controller |       | base64 -d)'
2026-03-14 09:44:27.302143 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.302147 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.302152 | controller |     - '          exit 1'
2026-03-14 09:44:27.302156 | controller |     - '        fi'
2026-03-14 09:44:27.302160 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.302164 | controller |     - '        '
2026-03-14 09:44:27.302168 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.302173 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.302177 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.302181 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.302185 | controller |       --timeout=300s'
2026-03-14 09:44:27.302199 | controller |     - '        '
2026-03-14 09:44:27.302203 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.302208 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.302212 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.302216 | controller |     - '        '
2026-03-14 09:44:27.302220 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.302224 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.302229 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.302233 | controller |     - '        '
2026-03-14 09:44:27.302243 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.302247 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.302252 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.302258 | controller |       available"'
2026-03-14 09:44:27.302264 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.302270 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.302279 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.302285 | controller |     - '          exit 1'
2026-03-14 09:44:27.302301 | controller |     - '        fi'
2026-03-14 09:44:27.302308 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.302313 | controller |     - '        '
2026-03-14 09:44:27.302317 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.302321 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.302325 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.302329 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.302334 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.302338 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.302342 | controller |       -- \'
2026-03-14 09:44:27.302346 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.302350 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.302355 | controller |     - '    logger.go:42: 09:30:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.302359 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.302363 | controller |     - '    logger.go:42: 09:30:01 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.302367 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.302372 | controller |       found'
2026-03-14 09:44:27.302376 | controller |     - '    logger.go:42: 09:30:02 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.302380 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.302384 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.302388 | controller |     - '        '
2026-03-14 09:44:27.302393 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.302397 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.302401 | controller |     - '        '
2026-03-14 09:44:27.302405 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.302409 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.302414 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.302418 | controller |     - '          exit 1'
2026-03-14 09:44:27.302422 | controller |     - '        fi'
2026-03-14 09:44:27.302426 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.302430 | controller |     - '        '
2026-03-14 09:44:27.302438 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.302442 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.302446 | controller |       | base64 -d)'
2026-03-14 09:44:27.302450 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.302455 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.302459 | controller |     - '          exit 1'
2026-03-14 09:44:27.302463 | controller |     - '        fi'
2026-03-14 09:44:27.302467 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.302471 | controller |     - '        '
2026-03-14 09:44:27.302476 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.302480 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.302484 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.302499 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.302503 | controller |       --timeout=300s'
2026-03-14 09:44:27.302507 | controller |     - '        '
2026-03-14 09:44:27.302511 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.302516 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.302520 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.302524 | controller |     - '        '
2026-03-14 09:44:27.302528 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.302533 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.302537 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.302541 | controller |     - '        '
2026-03-14 09:44:27.302551 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.302559 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.302564 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.302568 | controller |       available"'
2026-03-14 09:44:27.302572 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.302576 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.302580 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.302585 | controller |     - '          exit 1'
2026-03-14 09:44:27.302589 | controller |     - '        fi'
2026-03-14 09:44:27.302593 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.302597 | controller |     - '        '
2026-03-14 09:44:27.302602 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.302614 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.302619 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.302623 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.302627 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.302631 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.302635 | controller |       -- \'
2026-03-14 09:44:27.302640 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.302644 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.302648 | controller |     - '    logger.go:42: 09:30:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.302652 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.302656 | controller |     - '    logger.go:42: 09:30:02 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.302660 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.302665 | controller |       found'
2026-03-14 09:44:27.302669 | controller |     - '    logger.go:42: 09:30:03 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.302673 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.302677 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.302681 | controller |     - '        '
2026-03-14 09:44:27.302686 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.302698 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.302702 | controller |     - '        '
2026-03-14 09:44:27.302706 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.302711 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.302715 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.302719 | controller |     - '          exit 1'
2026-03-14 09:44:27.302723 | controller |     - '        fi'
2026-03-14 09:44:27.302728 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.302732 | controller |     - '        '
2026-03-14 09:44:27.302736 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.302740 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.302744 | controller |       | base64 -d)'
2026-03-14 09:44:27.302749 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.302753 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.302757 | controller |     - '          exit 1'
2026-03-14 09:44:27.302761 | controller |     - '        fi'
2026-03-14 09:44:27.302765 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.302770 | controller |     - '        '
2026-03-14 09:44:27.302774 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.302778 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.302782 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.302786 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.302791 | controller |       --timeout=300s'
2026-03-14 09:44:27.302795 | controller |     - '        '
2026-03-14 09:44:27.302799 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.302803 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.302807 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.302812 | controller |     - '        '
2026-03-14 09:44:27.302816 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.302820 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.302824 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.302828 | controller |     - '        '
2026-03-14 09:44:27.302833 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.302843 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.302847 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.302852 | controller |       available"'
2026-03-14 09:44:27.302856 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.302860 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.302864 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.302869 | controller |     - '          exit 1'
2026-03-14 09:44:27.302873 | controller |     - '        fi'
2026-03-14 09:44:27.302877 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.302881 | controller |     - '        '
2026-03-14 09:44:27.302886 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.302890 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.302896 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.302900 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.302904 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.302909 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.302913 | controller |       -- \'
2026-03-14 09:44:27.302917 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.302921 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.302925 | controller |     - '    logger.go:42: 09:30:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.302930 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.302934 | controller |     - '    logger.go:42: 09:30:03 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.302938 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.302942 | controller |       found'
2026-03-14 09:44:27.302946 | controller |     - '    logger.go:42: 09:30:04 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.302951 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.302955 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.302959 | controller |     - '        '
2026-03-14 09:44:27.302963 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.302968 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.302974 | controller |     - '        '
2026-03-14 09:44:27.302979 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.302983 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.302987 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.302991 | controller |     - '          exit 1'
2026-03-14 09:44:27.302995 | controller |     - '        fi'
2026-03-14 09:44:27.303000 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.303004 | controller |     - '        '
2026-03-14 09:44:27.303008 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.303012 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.303016 | controller |       | base64 -d)'
2026-03-14 09:44:27.303021 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.303025 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.303029 | controller |     - '          exit 1'
2026-03-14 09:44:27.303033 | controller |     - '        fi'
2026-03-14 09:44:27.303039 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.303043 | controller |     - '        '
2026-03-14 09:44:27.303048 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.303052 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.303056 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.303060 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.303064 | controller |       --timeout=300s'
2026-03-14 09:44:27.303069 | controller |     - '        '
2026-03-14 09:44:27.303073 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.303077 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.303081 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.303087 | controller |     - '        '
2026-03-14 09:44:27.303092 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.303096 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.303100 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.303104 | controller |     - '        '
2026-03-14 09:44:27.303108 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.303113 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.303123 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.303127 | controller |       available"'
2026-03-14 09:44:27.303132 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.303136 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.303140 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.303144 | controller |     - '          exit 1'
2026-03-14 09:44:27.303148 | controller |     - '        fi'
2026-03-14 09:44:27.303153 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.303157 | controller |     - '        '
2026-03-14 09:44:27.303161 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.303165 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.303169 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.303183 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.303187 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.303192 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.303196 | controller |       -- \'
2026-03-14 09:44:27.303201 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.303205 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.303211 | controller |     - '    logger.go:42: 09:30:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.303215 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.303219 | controller |     - '    logger.go:42: 09:30:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.303224 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.303228 | controller |       found'
2026-03-14 09:44:27.303232 | controller |     - '    logger.go:42: 09:30:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.303236 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.303240 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.303245 | controller |     - '        '
2026-03-14 09:44:27.303249 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.303254 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.303260 | controller |     - '        '
2026-03-14 09:44:27.303266 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.303272 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.303278 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.303284 | controller |     - '          exit 1'
2026-03-14 09:44:27.303302 | controller |     - '        fi'
2026-03-14 09:44:27.303311 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.303316 | controller |     - '        '
2026-03-14 09:44:27.303320 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.303324 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.303328 | controller |       | base64 -d)'
2026-03-14 09:44:27.303332 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.303337 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.303341 | controller |     - '          exit 1'
2026-03-14 09:44:27.303345 | controller |     - '        fi'
2026-03-14 09:44:27.303349 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.303354 | controller |     - '        '
2026-03-14 09:44:27.303358 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.303362 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.303366 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.303370 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.303374 | controller |       --timeout=300s'
2026-03-14 09:44:27.303379 | controller |     - '        '
2026-03-14 09:44:27.303383 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.303387 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.303391 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.303395 | controller |     - '        '
2026-03-14 09:44:27.303400 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.303404 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.303408 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.303412 | controller |     - '        '
2026-03-14 09:44:27.303417 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.303421 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.303432 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.303436 | controller |       available"'
2026-03-14 09:44:27.303440 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.303445 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.303449 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.303453 | controller |     - '          exit 1'
2026-03-14 09:44:27.303457 | controller |     - '        fi'
2026-03-14 09:44:27.303461 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.303466 | controller |     - '        '
2026-03-14 09:44:27.303470 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.303474 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.303478 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.303482 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.303487 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.303491 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.303495 | controller |       -- \'
2026-03-14 09:44:27.303501 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.303506 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.303510 | controller |     - '    logger.go:42: 09:30:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.303514 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.303518 | controller |     - '    logger.go:42: 09:30:06 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.303522 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.303527 | controller |       found'
2026-03-14 09:44:27.303531 | controller |     - '    logger.go:42: 09:30:07 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.303535 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.303539 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.303544 | controller |     - '        '
2026-03-14 09:44:27.303548 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.303552 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.303556 | controller |     - '        '
2026-03-14 09:44:27.303560 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.303565 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.303569 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.303573 | controller |     - '          exit 1'
2026-03-14 09:44:27.303577 | controller |     - '        fi'
2026-03-14 09:44:27.303581 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.303586 | controller |     - '        '
2026-03-14 09:44:27.303590 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.303594 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.303598 | controller |       | base64 -d)'
2026-03-14 09:44:27.303603 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.303607 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.303611 | controller |     - '          exit 1'
2026-03-14 09:44:27.303615 | controller |     - '        fi'
2026-03-14 09:44:27.303619 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.303623 | controller |     - '        '
2026-03-14 09:44:27.303628 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.303632 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.303636 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.303640 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.303644 | controller |       --timeout=300s'
2026-03-14 09:44:27.303649 | controller |     - '        '
2026-03-14 09:44:27.303653 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.303657 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.303664 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.303668 | controller |     - '        '
2026-03-14 09:44:27.303672 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.303677 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.303681 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.303685 | controller |     - '        '
2026-03-14 09:44:27.303692 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.303696 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.303700 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.303704 | controller |       available"'
2026-03-14 09:44:27.303714 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.303721 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.303725 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.303729 | controller |     - '          exit 1'
2026-03-14 09:44:27.303733 | controller |     - '        fi'
2026-03-14 09:44:27.303738 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.303742 | controller |     - '        '
2026-03-14 09:44:27.303746 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.303750 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.303754 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.303759 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.303763 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.303767 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.303771 | controller |       -- \'
2026-03-14 09:44:27.303775 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.303782 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.303787 | controller |     - '    logger.go:42: 09:30:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.303791 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.303795 | controller |     - '    logger.go:42: 09:30:07 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.303799 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.303803 | controller |       found'
2026-03-14 09:44:27.303808 | controller |     - '    logger.go:42: 09:30:08 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.303812 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.303816 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.303820 | controller |     - '        '
2026-03-14 09:44:27.303824 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.303829 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.303833 | controller |     - '        '
2026-03-14 09:44:27.303837 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.303841 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.303847 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.303851 | controller |     - '          exit 1'
2026-03-14 09:44:27.303856 | controller |     - '        fi'
2026-03-14 09:44:27.303860 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.303864 | controller |     - '        '
2026-03-14 09:44:27.303868 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.303873 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.303877 | controller |       | base64 -d)'
2026-03-14 09:44:27.303881 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.303886 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.303903 | controller |     - '          exit 1'
2026-03-14 09:44:27.303909 | controller |     - '        fi'
2026-03-14 09:44:27.303915 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.303921 | controller |     - '        '
2026-03-14 09:44:27.303927 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.303932 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.303938 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.303944 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.303950 | controller |       --timeout=300s'
2026-03-14 09:44:27.303955 | controller |     - '        '
2026-03-14 09:44:27.303961 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.303966 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.303972 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.303976 | controller |     - '        '
2026-03-14 09:44:27.303980 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.303985 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.303989 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.303993 | controller |     - '        '
2026-03-14 09:44:27.303997 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.304002 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.304006 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.304010 | controller |       available"'
2026-03-14 09:44:27.304014 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.304026 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.304031 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.304035 | controller |     - '          exit 1'
2026-03-14 09:44:27.304039 | controller |     - '        fi'
2026-03-14 09:44:27.304043 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.304048 | controller |     - '        '
2026-03-14 09:44:27.304052 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.304056 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.304060 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.304065 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.304069 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.304073 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.304077 | controller |       -- \'
2026-03-14 09:44:27.304082 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.304086 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.304090 | controller |     - '    logger.go:42: 09:30:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.304094 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.304099 | controller |     - '    logger.go:42: 09:30:08 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.304106 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.304110 | controller |       found'
2026-03-14 09:44:27.304114 | controller |     - '    logger.go:42: 09:30:09 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.304119 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.304123 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.304127 | controller |     - '        '
2026-03-14 09:44:27.304132 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.304136 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.304140 | controller |     - '        '
2026-03-14 09:44:27.304144 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.304148 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.304153 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.304157 | controller |     - '          exit 1'
2026-03-14 09:44:27.304164 | controller |     - '        fi'
2026-03-14 09:44:27.304168 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.304172 | controller |     - '        '
2026-03-14 09:44:27.304176 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.304180 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.304185 | controller |       | base64 -d)'
2026-03-14 09:44:27.304189 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.304193 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.304197 | controller |     - '          exit 1'
2026-03-14 09:44:27.304201 | controller |     - '        fi'
2026-03-14 09:44:27.304206 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.304210 | controller |     - '        '
2026-03-14 09:44:27.304214 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.304218 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.304222 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.304227 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.304231 | controller |       --timeout=300s'
2026-03-14 09:44:27.304235 | controller |     - '        '
2026-03-14 09:44:27.304239 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.304243 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.304248 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.304253 | controller |     - '        '
2026-03-14 09:44:27.304259 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.304265 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.304271 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.304277 | controller |     - '        '
2026-03-14 09:44:27.304282 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.304289 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.304308 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.304313 | controller |       available"'
2026-03-14 09:44:27.304317 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.304328 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.304332 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.304339 | controller |     - '          exit 1'
2026-03-14 09:44:27.304343 | controller |     - '        fi'
2026-03-14 09:44:27.304347 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.304352 | controller |     - '        '
2026-03-14 09:44:27.304356 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.304360 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.304364 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.304368 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.304373 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.304377 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.304381 | controller |       -- \'
2026-03-14 09:44:27.304385 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.304389 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.304394 | controller |     - '    logger.go:42: 09:30:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.304398 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.304404 | controller |     - '    logger.go:42: 09:30:09 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.304408 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.304413 | controller |       found'
2026-03-14 09:44:27.304417 | controller |     - '    logger.go:42: 09:30:10 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.304421 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.304425 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.304430 | controller |     - '        '
2026-03-14 09:44:27.304434 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.304438 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.304442 | controller |     - '        '
2026-03-14 09:44:27.304446 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.304450 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.304455 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.304459 | controller |     - '          exit 1'
2026-03-14 09:44:27.304463 | controller |     - '        fi'
2026-03-14 09:44:27.304467 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.304471 | controller |     - '        '
2026-03-14 09:44:27.304476 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.304480 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.304484 | controller |       | base64 -d)'
2026-03-14 09:44:27.304488 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.304492 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.304497 | controller |     - '          exit 1'
2026-03-14 09:44:27.304501 | controller |     - '        fi'
2026-03-14 09:44:27.304505 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.304509 | controller |     - '        '
2026-03-14 09:44:27.304513 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.304518 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.304522 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.304526 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.304533 | controller |       --timeout=300s'
2026-03-14 09:44:27.304537 | controller |     - '        '
2026-03-14 09:44:27.304541 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.304546 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.304550 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.304554 | controller |     - '        '
2026-03-14 09:44:27.304560 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.304564 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.304568 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.304572 | controller |     - '        '
2026-03-14 09:44:27.304577 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.304581 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.304585 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.304589 | controller |       available"'
2026-03-14 09:44:27.304593 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.304598 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.304608 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.304612 | controller |     - '          exit 1'
2026-03-14 09:44:27.304617 | controller |     - '        fi'
2026-03-14 09:44:27.304621 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.304625 | controller |     - '        '
2026-03-14 09:44:27.304629 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.304633 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.304638 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.304643 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.304649 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.304653 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.304657 | controller |       -- \'
2026-03-14 09:44:27.304661 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.304666 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.304670 | controller |     - '    logger.go:42: 09:30:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.304674 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.304678 | controller |     - '    logger.go:42: 09:30:10 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.304682 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.304687 | controller |       found'
2026-03-14 09:44:27.304691 | controller |     - '    logger.go:42: 09:30:11 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.304695 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.304699 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.304703 | controller |     - '        '
2026-03-14 09:44:27.304708 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.304712 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.304718 | controller |     - '        '
2026-03-14 09:44:27.304722 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.304726 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.304730 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.304735 | controller |     - '          exit 1'
2026-03-14 09:44:27.304740 | controller |     - '        fi'
2026-03-14 09:44:27.304746 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.304751 | controller |     - '        '
2026-03-14 09:44:27.304757 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.304762 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.304766 | controller |       | base64 -d)'
2026-03-14 09:44:27.304771 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.304775 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.304779 | controller |     - '          exit 1'
2026-03-14 09:44:27.304783 | controller |     - '        fi'
2026-03-14 09:44:27.304788 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.304792 | controller |     - '        '
2026-03-14 09:44:27.304796 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.304800 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.304804 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.304809 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.304813 | controller |       --timeout=300s'
2026-03-14 09:44:27.304817 | controller |     - '        '
2026-03-14 09:44:27.304821 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.304825 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.304830 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.304834 | controller |     - '        '
2026-03-14 09:44:27.304838 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.304842 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.304846 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.304850 | controller |     - '        '
2026-03-14 09:44:27.304855 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.304859 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.304863 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.304867 | controller |       available"'
2026-03-14 09:44:27.304872 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.304876 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.304880 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.304890 | controller |     - '          exit 1'
2026-03-14 09:44:27.304895 | controller |     - '        fi'
2026-03-14 09:44:27.304899 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.304903 | controller |     - '        '
2026-03-14 09:44:27.304908 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.304912 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.304916 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.304922 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.304927 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.304931 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.304935 | controller |       -- \'
2026-03-14 09:44:27.304939 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.304943 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.304948 | controller |     - '    logger.go:42: 09:30:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.304952 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.304956 | controller |     - '    logger.go:42: 09:30:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.304962 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.304967 | controller |       found'
2026-03-14 09:44:27.304971 | controller |     - '    logger.go:42: 09:30:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.304975 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.304979 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.304983 | controller |     - '        '
2026-03-14 09:44:27.304988 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.304992 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.304996 | controller |     - '        '
2026-03-14 09:44:27.305000 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.305004 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.305009 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.305013 | controller |     - '          exit 1'
2026-03-14 09:44:27.305017 | controller |     - '        fi'
2026-03-14 09:44:27.305021 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.305026 | controller |     - '        '
2026-03-14 09:44:27.305031 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.305036 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.305040 | controller |       | base64 -d)'
2026-03-14 09:44:27.305045 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.305049 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.305053 | controller |     - '          exit 1'
2026-03-14 09:44:27.305058 | controller |     - '        fi'
2026-03-14 09:44:27.305062 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.305066 | controller |     - '        '
2026-03-14 09:44:27.305070 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.305074 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.305078 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.305083 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.305087 | controller |       --timeout=300s'
2026-03-14 09:44:27.305091 | controller |     - '        '
2026-03-14 09:44:27.305095 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.305099 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.305104 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.305110 | controller |     - '        '
2026-03-14 09:44:27.305114 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.305118 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.305122 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.305127 | controller |     - '        '
2026-03-14 09:44:27.305131 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.305135 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.305139 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.305144 | controller |       available"'
2026-03-14 09:44:27.305148 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.305152 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.305156 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.305161 | controller |     - '          exit 1'
2026-03-14 09:44:27.305168 | controller |     - '        fi'
2026-03-14 09:44:27.305178 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.305183 | controller |     - '        '
2026-03-14 09:44:27.305187 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.305191 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.305196 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.305200 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.305204 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.305208 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.305213 | controller |       -- \'
2026-03-14 09:44:27.305217 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.305221 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.305225 | controller |     - '    logger.go:42: 09:30:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.305229 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.305233 | controller |     - '    logger.go:42: 09:30:13 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.305238 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.305242 | controller |       found'
2026-03-14 09:44:27.305246 | controller |     - '    logger.go:42: 09:30:14 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.305251 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.305256 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.305262 | controller |     - '        '
2026-03-14 09:44:27.305268 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.305274 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.305280 | controller |     - '        '
2026-03-14 09:44:27.305285 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.305304 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.305310 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.305315 | controller |     - '          exit 1'
2026-03-14 09:44:27.305319 | controller |     - '        fi'
2026-03-14 09:44:27.305323 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.305330 | controller |     - '        '
2026-03-14 09:44:27.305334 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.305338 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.305343 | controller |       | base64 -d)'
2026-03-14 09:44:27.305347 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.305351 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.305355 | controller |     - '          exit 1'
2026-03-14 09:44:27.305360 | controller |     - '        fi'
2026-03-14 09:44:27.305364 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.305371 | controller |     - '        '
2026-03-14 09:44:27.305375 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.305379 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.305383 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.305388 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.305392 | controller |       --timeout=300s'
2026-03-14 09:44:27.305396 | controller |     - '        '
2026-03-14 09:44:27.305400 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.305404 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.305409 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.305413 | controller |     - '        '
2026-03-14 09:44:27.305417 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.305421 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.305425 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.305430 | controller |     - '        '
2026-03-14 09:44:27.305434 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.305438 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.305442 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.305446 | controller |       available"'
2026-03-14 09:44:27.305451 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.305455 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.305459 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.305463 | controller |     - '          exit 1'
2026-03-14 09:44:27.305467 | controller |     - '        fi'
2026-03-14 09:44:27.305478 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.305483 | controller |     - '        '
2026-03-14 09:44:27.305487 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.305492 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.305496 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.305500 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.305504 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.305508 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.305513 | controller |       -- \'
2026-03-14 09:44:27.305517 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.305523 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.305527 | controller |     - '    logger.go:42: 09:30:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.305531 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.305536 | controller |     - '    logger.go:42: 09:30:14 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.305540 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.305544 | controller |       found'
2026-03-14 09:44:27.305551 | controller |     - '    logger.go:42: 09:30:15 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.305555 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.305559 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.305564 | controller |     - '        '
2026-03-14 09:44:27.305568 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.305572 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.305576 | controller |     - '        '
2026-03-14 09:44:27.305580 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.305584 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.305589 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.305593 | controller |     - '          exit 1'
2026-03-14 09:44:27.305597 | controller |     - '        fi'
2026-03-14 09:44:27.305601 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.305605 | controller |     - '        '
2026-03-14 09:44:27.305610 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.305614 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.305618 | controller |       | base64 -d)'
2026-03-14 09:44:27.305622 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.305627 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.305631 | controller |     - '          exit 1'
2026-03-14 09:44:27.305635 | controller |     - '        fi'
2026-03-14 09:44:27.305639 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.305643 | controller |     - '        '
2026-03-14 09:44:27.305648 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.305652 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.305656 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.305660 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.305664 | controller |       --timeout=300s'
2026-03-14 09:44:27.305669 | controller |     - '        '
2026-03-14 09:44:27.305674 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.305678 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.305682 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.305686 | controller |     - '        '
2026-03-14 09:44:27.305690 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.305696 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.305700 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.305705 | controller |     - '        '
2026-03-14 09:44:27.305709 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.305715 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.305719 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.305723 | controller |       available"'
2026-03-14 09:44:27.305727 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.305732 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.305736 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.305740 | controller |     - '          exit 1'
2026-03-14 09:44:27.305744 | controller |     - '        fi'
2026-03-14 09:44:27.305749 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.305753 | controller |     - '        '
2026-03-14 09:44:27.305763 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.305768 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.305772 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.305776 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.305780 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.305785 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.305789 | controller |       -- \'
2026-03-14 09:44:27.305793 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.305797 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.305802 | controller |     - '    logger.go:42: 09:30:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.305806 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.305810 | controller |     - '    logger.go:42: 09:30:15 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.305814 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.305819 | controller |       found'
2026-03-14 09:44:27.305823 | controller |     - '    logger.go:42: 09:30:16 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.305827 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.305831 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.305835 | controller |     - '        '
2026-03-14 09:44:27.305840 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.305844 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.305848 | controller |     - '        '
2026-03-14 09:44:27.305852 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.305857 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.305861 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.305865 | controller |     - '          exit 1'
2026-03-14 09:44:27.305869 | controller |     - '        fi'
2026-03-14 09:44:27.305874 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.305878 | controller |     - '        '
2026-03-14 09:44:27.305882 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.305886 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.305890 | controller |       | base64 -d)'
2026-03-14 09:44:27.305895 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.305901 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.305905 | controller |     - '          exit 1'
2026-03-14 09:44:27.305911 | controller |     - '        fi'
2026-03-14 09:44:27.305935 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.305940 | controller |     - '        '
2026-03-14 09:44:27.305945 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.305949 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.305953 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.305957 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.305962 | controller |       --timeout=300s'
2026-03-14 09:44:27.305966 | controller |     - '        '
2026-03-14 09:44:27.305970 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.305974 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.305979 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.305983 | controller |     - '        '
2026-03-14 09:44:27.305987 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.305991 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.305996 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.306000 | controller |     - '        '
2026-03-14 09:44:27.306004 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.306008 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.306012 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.306017 | controller |       available"'
2026-03-14 09:44:27.306021 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.306025 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.306029 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.306033 | controller |     - '          exit 1'
2026-03-14 09:44:27.306038 | controller |     - '        fi'
2026-03-14 09:44:27.306042 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.306046 | controller |     - '        '
2026-03-14 09:44:27.306056 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.306061 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.306065 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.306069 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.306073 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.306077 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.306082 | controller |       -- \'
2026-03-14 09:44:27.306086 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.306090 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.306094 | controller |     - '    logger.go:42: 09:30:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.306099 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.306103 | controller |     - '    logger.go:42: 09:30:16 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.306107 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.306113 | controller |       found'
2026-03-14 09:44:27.306118 | controller |     - '    logger.go:42: 09:30:17 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.306122 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.306126 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.306130 | controller |     - '        '
2026-03-14 09:44:27.306137 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.306141 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.306145 | controller |     - '        '
2026-03-14 09:44:27.306149 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.306154 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.306158 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.306162 | controller |     - '          exit 1'
2026-03-14 09:44:27.306166 | controller |     - '        fi'
2026-03-14 09:44:27.306171 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.306175 | controller |     - '        '
2026-03-14 09:44:27.306179 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.306183 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.306187 | controller |       | base64 -d)'
2026-03-14 09:44:27.306192 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.306196 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.306200 | controller |     - '          exit 1'
2026-03-14 09:44:27.306204 | controller |     - '        fi'
2026-03-14 09:44:27.306209 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.306213 | controller |     - '        '
2026-03-14 09:44:27.306217 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.306221 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.306225 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.306230 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.306234 | controller |       --timeout=300s'
2026-03-14 09:44:27.306241 | controller |     - '        '
2026-03-14 09:44:27.306245 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.306250 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.306255 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.306261 | controller |     - '        '
2026-03-14 09:44:27.306267 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.306273 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.306279 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.306284 | controller |     - '        '
2026-03-14 09:44:27.306302 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.306309 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.306314 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.306318 | controller |       available"'
2026-03-14 09:44:27.306322 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.306327 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.306331 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.306337 | controller |     - '          exit 1'
2026-03-14 09:44:27.306344 | controller |     - '        fi'
2026-03-14 09:44:27.306349 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.306353 | controller |     - '        '
2026-03-14 09:44:27.306357 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.306369 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.306373 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.306377 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.306382 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.306386 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.306390 | controller |       -- \'
2026-03-14 09:44:27.306394 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.306399 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.306403 | controller |     - '    logger.go:42: 09:30:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.306407 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.306411 | controller |     - '    logger.go:42: 09:30:17 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.306416 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.306420 | controller |       found'
2026-03-14 09:44:27.306424 | controller |     - '    logger.go:42: 09:30:18 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.306428 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.306432 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.306437 | controller |     - '        '
2026-03-14 09:44:27.306441 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.306445 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.306449 | controller |     - '        '
2026-03-14 09:44:27.306453 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.306458 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.306462 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.306466 | controller |     - '          exit 1'
2026-03-14 09:44:27.306470 | controller |     - '        fi'
2026-03-14 09:44:27.306475 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.306479 | controller |     - '        '
2026-03-14 09:44:27.306483 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.306487 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.306491 | controller |       | base64 -d)'
2026-03-14 09:44:27.306496 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.306500 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.306504 | controller |     - '          exit 1'
2026-03-14 09:44:27.306508 | controller |     - '        fi'
2026-03-14 09:44:27.306512 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.306517 | controller |     - '        '
2026-03-14 09:44:27.306521 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.306525 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.306529 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.306533 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.306541 | controller |       --timeout=300s'
2026-03-14 09:44:27.306545 | controller |     - '        '
2026-03-14 09:44:27.306549 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.306553 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.306557 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.306562 | controller |     - '        '
2026-03-14 09:44:27.306566 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.306570 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.306574 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.306578 | controller |     - '        '
2026-03-14 09:44:27.306583 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.306587 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.306591 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.306595 | controller |       available"'
2026-03-14 09:44:27.306599 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.306604 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.306608 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.306612 | controller |     - '          exit 1'
2026-03-14 09:44:27.306616 | controller |     - '        fi'
2026-03-14 09:44:27.306620 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.306624 | controller |     - '        '
2026-03-14 09:44:27.306629 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.306633 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.306643 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.306648 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.306652 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.306657 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.306663 | controller |       -- \'
2026-03-14 09:44:27.306667 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.306671 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.306675 | controller |     - '    logger.go:42: 09:30:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.306680 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.306684 | controller |     - '    logger.go:42: 09:30:18 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.306688 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.306692 | controller |       found'
2026-03-14 09:44:27.306696 | controller |     - '    logger.go:42: 09:30:19 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.306701 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.306705 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.306709 | controller |     - '        '
2026-03-14 09:44:27.306713 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.306719 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.306725 | controller |     - '        '
2026-03-14 09:44:27.306730 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.306734 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.306738 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.306742 | controller |     - '          exit 1'
2026-03-14 09:44:27.306747 | controller |     - '        fi'
2026-03-14 09:44:27.306751 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.306755 | controller |     - '        '
2026-03-14 09:44:27.306759 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.306763 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.306768 | controller |       | base64 -d)'
2026-03-14 09:44:27.306772 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.306776 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.306780 | controller |     - '          exit 1'
2026-03-14 09:44:27.306784 | controller |     - '        fi'
2026-03-14 09:44:27.306789 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.306793 | controller |     - '        '
2026-03-14 09:44:27.306797 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.306801 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.306805 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.306810 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.306814 | controller |       --timeout=300s'
2026-03-14 09:44:27.306818 | controller |     - '        '
2026-03-14 09:44:27.306822 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.306826 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.306831 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.306835 | controller |     - '        '
2026-03-14 09:44:27.306839 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.306843 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.306847 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.306851 | controller |     - '        '
2026-03-14 09:44:27.306856 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.306860 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.306864 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.306868 | controller |       available"'
2026-03-14 09:44:27.306873 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.306877 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.306881 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.306885 | controller |     - '          exit 1'
2026-03-14 09:44:27.306889 | controller |     - '        fi'
2026-03-14 09:44:27.306893 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.306898 | controller |     - '        '
2026-03-14 09:44:27.306902 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.306906 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.306915 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.306921 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.306926 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.306930 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.306934 | controller |       -- \'
2026-03-14 09:44:27.306940 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.306945 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.306951 | controller |     - '    logger.go:42: 09:30:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.306957 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.306962 | controller |     - '    logger.go:42: 09:30:19 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.306968 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.306973 | controller |       found'
2026-03-14 09:44:27.306979 | controller |     - '    logger.go:42: 09:30:20 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.306985 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.306990 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.306996 | controller |     - '        '
2026-03-14 09:44:27.307001 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.307005 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.307009 | controller |     - '        '
2026-03-14 09:44:27.307013 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.307017 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.307022 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.307026 | controller |     - '          exit 1'
2026-03-14 09:44:27.307030 | controller |     - '        fi'
2026-03-14 09:44:27.307034 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.307038 | controller |     - '        '
2026-03-14 09:44:27.307042 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.307047 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.307051 | controller |       | base64 -d)'
2026-03-14 09:44:27.307055 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.307059 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.307063 | controller |     - '          exit 1'
2026-03-14 09:44:27.307068 | controller |     - '        fi'
2026-03-14 09:44:27.307072 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.307076 | controller |     - '        '
2026-03-14 09:44:27.307080 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.307084 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.307089 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.307093 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.307097 | controller |       --timeout=300s'
2026-03-14 09:44:27.307101 | controller |     - '        '
2026-03-14 09:44:27.307105 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.307109 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.307114 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.307118 | controller |     - '        '
2026-03-14 09:44:27.307124 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.307143 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.307147 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.307151 | controller |     - '        '
2026-03-14 09:44:27.307155 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.307160 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.307164 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.307168 | controller |       available"'
2026-03-14 09:44:27.307172 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.307177 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.307181 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.307185 | controller |     - '          exit 1'
2026-03-14 09:44:27.307189 | controller |     - '        fi'
2026-03-14 09:44:27.307193 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.307198 | controller |     - '        '
2026-03-14 09:44:27.307202 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.307206 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.307218 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.307222 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.307226 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.307231 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.307235 | controller |       -- \'
2026-03-14 09:44:27.307239 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.307243 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.307251 | controller |     - '    logger.go:42: 09:30:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.307257 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.307263 | controller |     - '    logger.go:42: 09:30:21 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.307269 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.307275 | controller |       found'
2026-03-14 09:44:27.307281 | controller |     - '    logger.go:42: 09:30:22 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.307300 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.307309 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.307315 | controller |     - '        '
2026-03-14 09:44:27.307320 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.307325 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.307330 | controller |     - '        '
2026-03-14 09:44:27.307337 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.307341 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.307345 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.307349 | controller |     - '          exit 1'
2026-03-14 09:44:27.307354 | controller |     - '        fi'
2026-03-14 09:44:27.307358 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.307362 | controller |     - '        '
2026-03-14 09:44:27.307390 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.307397 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.307402 | controller |       | base64 -d)'
2026-03-14 09:44:27.307407 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.307411 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.307415 | controller |     - '          exit 1'
2026-03-14 09:44:27.307419 | controller |     - '        fi'
2026-03-14 09:44:27.307423 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.307428 | controller |     - '        '
2026-03-14 09:44:27.307432 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.307436 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.307440 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.307444 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.307448 | controller |       --timeout=300s'
2026-03-14 09:44:27.307453 | controller |     - '        '
2026-03-14 09:44:27.307457 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.307461 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.307465 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.307469 | controller |     - '        '
2026-03-14 09:44:27.307509 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.307514 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.307518 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.307522 | controller |     - '        '
2026-03-14 09:44:27.307526 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.307531 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.307535 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.307546 | controller |       available"'
2026-03-14 09:44:27.307550 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.307554 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.307558 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.307563 | controller |     - '          exit 1'
2026-03-14 09:44:27.307567 | controller |     - '        fi'
2026-03-14 09:44:27.307571 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.307575 | controller |     - '        '
2026-03-14 09:44:27.307579 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.307583 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.307588 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.307599 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.307604 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.307608 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.307612 | controller |       -- \'
2026-03-14 09:44:27.307617 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.307626 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.307645 | controller |     - '    logger.go:42: 09:30:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.307653 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.307659 | controller |     - '    logger.go:42: 09:30:22 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.307665 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.307671 | controller |       found'
2026-03-14 09:44:27.307677 | controller |     - '    logger.go:42: 09:30:23 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.307683 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.307694 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.307700 | controller |     - '        '
2026-03-14 09:44:27.307706 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.307712 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.307717 | controller |     - '        '
2026-03-14 09:44:27.307722 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.307728 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.307733 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.307739 | controller |     - '          exit 1'
2026-03-14 09:44:27.307745 | controller |     - '        fi'
2026-03-14 09:44:27.307751 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.307755 | controller |     - '        '
2026-03-14 09:44:27.307762 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.307766 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.307770 | controller |       | base64 -d)'
2026-03-14 09:44:27.307775 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.307779 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.307783 | controller |     - '          exit 1'
2026-03-14 09:44:27.307787 | controller |     - '        fi'
2026-03-14 09:44:27.307791 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.307796 | controller |     - '        '
2026-03-14 09:44:27.307800 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.307806 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.307811 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.307817 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.307823 | controller |       --timeout=300s'
2026-03-14 09:44:27.307829 | controller |     - '        '
2026-03-14 09:44:27.307834 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.307840 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.307846 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.307852 | controller |     - '        '
2026-03-14 09:44:27.307857 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.307863 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.307870 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.307874 | controller |     - '        '
2026-03-14 09:44:27.307879 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.307885 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.307894 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.307898 | controller |       available"'
2026-03-14 09:44:27.307903 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.307907 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.307911 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.307915 | controller |     - '          exit 1'
2026-03-14 09:44:27.307919 | controller |     - '        fi'
2026-03-14 09:44:27.307924 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.307928 | controller |     - '        '
2026-03-14 09:44:27.307932 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.307936 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.307940 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.307944 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.307960 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.307964 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.307969 | controller |       -- \'
2026-03-14 09:44:27.307973 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.307977 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.307981 | controller |     - '    logger.go:42: 09:30:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.307986 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.307990 | controller |     - '    logger.go:42: 09:30:23 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.307994 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.307998 | controller |       found'
2026-03-14 09:44:27.308002 | controller |     - '    logger.go:42: 09:30:24 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.308006 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.308011 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.308015 | controller |     - '        '
2026-03-14 09:44:27.308019 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.308023 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.308027 | controller |     - '        '
2026-03-14 09:44:27.308032 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.308036 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.308040 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.308044 | controller |     - '          exit 1'
2026-03-14 09:44:27.308048 | controller |     - '        fi'
2026-03-14 09:44:27.308053 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.308059 | controller |     - '        '
2026-03-14 09:44:27.308063 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.308067 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.308071 | controller |       | base64 -d)'
2026-03-14 09:44:27.308076 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.308080 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.308084 | controller |     - '          exit 1'
2026-03-14 09:44:27.308088 | controller |     - '        fi'
2026-03-14 09:44:27.308092 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.308098 | controller |     - '        '
2026-03-14 09:44:27.308103 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.308107 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.308111 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.308115 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.308119 | controller |       --timeout=300s'
2026-03-14 09:44:27.308124 | controller |     - '        '
2026-03-14 09:44:27.308128 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.308132 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.308136 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.308141 | controller |     - '        '
2026-03-14 09:44:27.308145 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.308149 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.308153 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.308157 | controller |     - '        '
2026-03-14 09:44:27.308162 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.308166 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.308170 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.308174 | controller |       available"'
2026-03-14 09:44:27.308178 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.308182 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.308187 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.308191 | controller |     - '          exit 1'
2026-03-14 09:44:27.308195 | controller |     - '        fi'
2026-03-14 09:44:27.308199 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.308203 | controller |     - '        '
2026-03-14 09:44:27.308208 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.308212 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.308216 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.308220 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.308230 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.308236 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.308241 | controller |       -- \'
2026-03-14 09:44:27.308247 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.308253 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.308259 | controller |     - '    logger.go:42: 09:30:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.308264 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.308270 | controller |     - '    logger.go:42: 09:30:24 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.308276 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.308281 | controller |       found'
2026-03-14 09:44:27.308313 | controller |     - '    logger.go:42: 09:30:25 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.308322 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.308328 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.308334 | controller |     - '        '
2026-03-14 09:44:27.308339 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.308345 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.308350 | controller |     - '        '
2026-03-14 09:44:27.308356 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.308362 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.308368 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.308373 | controller |     - '          exit 1'
2026-03-14 09:44:27.308378 | controller |     - '        fi'
2026-03-14 09:44:27.308384 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.308389 | controller |     - '        '
2026-03-14 09:44:27.308394 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.308400 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.308406 | controller |       | base64 -d)'
2026-03-14 09:44:27.308411 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.308417 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.308423 | controller |     - '          exit 1'
2026-03-14 09:44:27.308429 | controller |     - '        fi'
2026-03-14 09:44:27.308435 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.308440 | controller |     - '        '
2026-03-14 09:44:27.308446 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.308452 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.308457 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.308463 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.308468 | controller |       --timeout=300s'
2026-03-14 09:44:27.308474 | controller |     - '        '
2026-03-14 09:44:27.308479 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.308485 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.308490 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.308495 | controller |     - '        '
2026-03-14 09:44:27.308501 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.308506 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.308512 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.308517 | controller |     - '        '
2026-03-14 09:44:27.308523 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.308528 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.308533 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.308539 | controller |       available"'
2026-03-14 09:44:27.308545 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.308550 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.308556 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.308580 | controller |     - '          exit 1'
2026-03-14 09:44:27.308589 | controller |     - '        fi'
2026-03-14 09:44:27.308595 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.308601 | controller |     - '        '
2026-03-14 09:44:27.308610 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.308615 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.308621 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.308627 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.308638 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.308643 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.308649 | controller |       -- \'
2026-03-14 09:44:27.308654 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.308659 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.308665 | controller |     - '    logger.go:42: 09:30:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.308671 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.308676 | controller |     - '    logger.go:42: 09:30:25 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.308682 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.308687 | controller |       found'
2026-03-14 09:44:27.308692 | controller |     - '    logger.go:42: 09:30:26 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.308698 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.308703 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.308709 | controller |     - '        '
2026-03-14 09:44:27.308714 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.308719 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.308725 | controller |     - '        '
2026-03-14 09:44:27.308730 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.308736 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.308741 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.308747 | controller |     - '          exit 1'
2026-03-14 09:44:27.308752 | controller |     - '        fi'
2026-03-14 09:44:27.308757 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.308763 | controller |     - '        '
2026-03-14 09:44:27.308768 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.308774 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.308779 | controller |       | base64 -d)'
2026-03-14 09:44:27.308784 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.308790 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.308795 | controller |     - '          exit 1'
2026-03-14 09:44:27.308801 | controller |     - '        fi'
2026-03-14 09:44:27.308806 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.308812 | controller |     - '        '
2026-03-14 09:44:27.308817 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.308822 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.308830 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.308836 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.308841 | controller |       --timeout=300s'
2026-03-14 09:44:27.308850 | controller |     - '        '
2026-03-14 09:44:27.308855 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.308860 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.308866 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.308871 | controller |     - '        '
2026-03-14 09:44:27.308878 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.308884 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.308889 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.308895 | controller |     - '        '
2026-03-14 09:44:27.308900 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.308906 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.308911 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.308917 | controller |       available"'
2026-03-14 09:44:27.308923 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.308928 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.308934 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.308940 | controller |     - '          exit 1'
2026-03-14 09:44:27.308945 | controller |     - '        fi'
2026-03-14 09:44:27.308951 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.308956 | controller |     - '        '
2026-03-14 09:44:27.308962 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.308967 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.308973 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.308978 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.308984 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.308998 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.309004 | controller |       -- \'
2026-03-14 09:44:27.309010 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.309015 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.309024 | controller |     - '    logger.go:42: 09:30:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.309029 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.309034 | controller |     - '    logger.go:42: 09:30:26 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.309040 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.309046 | controller |       found'
2026-03-14 09:44:27.309052 | controller |     - '    logger.go:42: 09:30:27 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.309057 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.309063 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.309069 | controller |     - '        '
2026-03-14 09:44:27.309075 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.309080 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.309085 | controller |     - '        '
2026-03-14 09:44:27.309091 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.309100 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.309106 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.309111 | controller |     - '          exit 1'
2026-03-14 09:44:27.309115 | controller |     - '        fi'
2026-03-14 09:44:27.309119 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.309125 | controller |     - '        '
2026-03-14 09:44:27.309131 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.309137 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.309142 | controller |       | base64 -d)'
2026-03-14 09:44:27.309148 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.309153 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.309159 | controller |     - '          exit 1'
2026-03-14 09:44:27.309165 | controller |     - '        fi'
2026-03-14 09:44:27.309170 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.309176 | controller |     - '        '
2026-03-14 09:44:27.309181 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.309187 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.309192 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.309198 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.309203 | controller |       --timeout=300s'
2026-03-14 09:44:27.309209 | controller |     - '        '
2026-03-14 09:44:27.309214 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.309220 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.309226 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.309232 | controller |     - '        '
2026-03-14 09:44:27.309238 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.309243 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.309247 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.309251 | controller |     - '        '
2026-03-14 09:44:27.309255 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.309259 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.309264 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.309268 | controller |       available"'
2026-03-14 09:44:27.309272 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.309279 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.309284 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.309306 | controller |     - '          exit 1'
2026-03-14 09:44:27.309314 | controller |     - '        fi'
2026-03-14 09:44:27.309318 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.309322 | controller |     - '        '
2026-03-14 09:44:27.309326 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.309330 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.309335 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.309339 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.309346 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.309350 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.309362 | controller |       -- \'
2026-03-14 09:44:27.309367 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.309371 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.309375 | controller |     - '    logger.go:42: 09:30:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.309380 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.309384 | controller |     - '    logger.go:42: 09:30:27 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.309388 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.309392 | controller |       found'
2026-03-14 09:44:27.309396 | controller |     - '    logger.go:42: 09:30:28 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.309401 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.309405 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.309409 | controller |     - '        '
2026-03-14 09:44:27.309413 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.309417 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.309422 | controller |     - '        '
2026-03-14 09:44:27.309426 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.309430 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.309434 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.309439 | controller |     - '          exit 1'
2026-03-14 09:44:27.309443 | controller |     - '        fi'
2026-03-14 09:44:27.309447 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.309451 | controller |     - '        '
2026-03-14 09:44:27.309455 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.309460 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.309464 | controller |       | base64 -d)'
2026-03-14 09:44:27.309468 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.309472 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.309476 | controller |     - '          exit 1'
2026-03-14 09:44:27.309480 | controller |     - '        fi'
2026-03-14 09:44:27.309485 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.309489 | controller |     - '        '
2026-03-14 09:44:27.309493 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.309497 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.309501 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.309508 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.309512 | controller |       --timeout=300s'
2026-03-14 09:44:27.309516 | controller |     - '        '
2026-03-14 09:44:27.309521 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.309525 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.309529 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.309533 | controller |     - '        '
2026-03-14 09:44:27.309537 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.309543 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.309548 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.309552 | controller |     - '        '
2026-03-14 09:44:27.309556 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.309560 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.309564 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.309568 | controller |       available"'
2026-03-14 09:44:27.309573 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.309577 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.309581 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.309585 | controller |     - '          exit 1'
2026-03-14 09:44:27.309589 | controller |     - '        fi'
2026-03-14 09:44:27.309594 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.309598 | controller |     - '        '
2026-03-14 09:44:27.309602 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.309606 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.309611 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.309616 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.309622 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.309628 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.309634 | controller |       -- \'
2026-03-14 09:44:27.309647 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.309652 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.309658 | controller |     - '    logger.go:42: 09:30:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.309664 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.309670 | controller |     - '    logger.go:42: 09:30:29 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.309676 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.309681 | controller |       found'
2026-03-14 09:44:27.309687 | controller |     - '    logger.go:42: 09:30:30 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.309693 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.309699 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.309705 | controller |     - '        '
2026-03-14 09:44:27.309710 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.309714 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.309718 | controller |     - '        '
2026-03-14 09:44:27.309723 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.309727 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.309731 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.309735 | controller |     - '          exit 1'
2026-03-14 09:44:27.309740 | controller |     - '        fi'
2026-03-14 09:44:27.309744 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.309748 | controller |     - '        '
2026-03-14 09:44:27.309752 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.309761 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.309765 | controller |       | base64 -d)'
2026-03-14 09:44:27.309769 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.309774 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.309778 | controller |     - '          exit 1'
2026-03-14 09:44:27.309782 | controller |     - '        fi'
2026-03-14 09:44:27.309786 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.309791 | controller |     - '        '
2026-03-14 09:44:27.309795 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.309799 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.309803 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.309807 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.309812 | controller |       --timeout=300s'
2026-03-14 09:44:27.309816 | controller |     - '        '
2026-03-14 09:44:27.309820 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.309824 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.309829 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.309833 | controller |     - '        '
2026-03-14 09:44:27.309840 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.309844 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.309848 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.309852 | controller |     - '        '
2026-03-14 09:44:27.309857 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.309861 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.309865 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.309869 | controller |       available"'
2026-03-14 09:44:27.309874 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.309878 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.309882 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.309887 | controller |     - '          exit 1'
2026-03-14 09:44:27.309893 | controller |     - '        fi'
2026-03-14 09:44:27.309899 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.309905 | controller |     - '        '
2026-03-14 09:44:27.309911 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.309917 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.309945 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.309952 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.309957 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.309962 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.309966 | controller |       -- \'
2026-03-14 09:44:27.309977 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.309982 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.309990 | controller |     - '    logger.go:42: 09:30:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.309994 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.309999 | controller |     - '    logger.go:42: 09:30:30 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.310003 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.310007 | controller |       found'
2026-03-14 09:44:27.310011 | controller |     - '    logger.go:42: 09:30:31 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.310016 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.310020 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.310024 | controller |     - '        '
2026-03-14 09:44:27.310028 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.310033 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.310037 | controller |     - '        '
2026-03-14 09:44:27.310041 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.310045 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.310049 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.310054 | controller |     - '          exit 1'
2026-03-14 09:44:27.310058 | controller |     - '        fi'
2026-03-14 09:44:27.310062 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.310066 | controller |     - '        '
2026-03-14 09:44:27.310070 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.310074 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.310079 | controller |       | base64 -d)'
2026-03-14 09:44:27.310083 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.310087 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.310091 | controller |     - '          exit 1'
2026-03-14 09:44:27.310095 | controller |     - '        fi'
2026-03-14 09:44:27.310100 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.310104 | controller |     - '        '
2026-03-14 09:44:27.310108 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.310112 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.310116 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.310121 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.310125 | controller |       --timeout=300s'
2026-03-14 09:44:27.310129 | controller |     - '        '
2026-03-14 09:44:27.310133 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.310137 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.310141 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.310146 | controller |     - '        '
2026-03-14 09:44:27.310150 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.310154 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.310158 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.310162 | controller |     - '        '
2026-03-14 09:44:27.310170 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.310175 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.310181 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.310185 | controller |       available"'
2026-03-14 09:44:27.310189 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.310194 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.310198 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.310202 | controller |     - '          exit 1'
2026-03-14 09:44:27.310206 | controller |     - '        fi'
2026-03-14 09:44:27.310210 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.310215 | controller |     - '        '
2026-03-14 09:44:27.310219 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.310223 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.310227 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.310231 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.310236 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.310240 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.310244 | controller |       -- \'
2026-03-14 09:44:27.310248 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.310258 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.310263 | controller |     - '    logger.go:42: 09:30:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.310267 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.310271 | controller |     - '    logger.go:42: 09:30:31 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.310275 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.310279 | controller |       found'
2026-03-14 09:44:27.310284 | controller |     - '    logger.go:42: 09:30:32 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.310301 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.310308 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.310313 | controller |     - '        '
2026-03-14 09:44:27.310317 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.310321 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.310325 | controller |     - '        '
2026-03-14 09:44:27.310329 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.310334 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.310338 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.310342 | controller |     - '          exit 1'
2026-03-14 09:44:27.310346 | controller |     - '        fi'
2026-03-14 09:44:27.310351 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.310355 | controller |     - '        '
2026-03-14 09:44:27.310359 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.310363 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.310367 | controller |       | base64 -d)'
2026-03-14 09:44:27.310372 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.310379 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.310383 | controller |     - '          exit 1'
2026-03-14 09:44:27.310387 | controller |     - '        fi'
2026-03-14 09:44:27.310391 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.310397 | controller |     - '        '
2026-03-14 09:44:27.310401 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.310406 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.310410 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.310414 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.310418 | controller |       --timeout=300s'
2026-03-14 09:44:27.310422 | controller |     - '        '
2026-03-14 09:44:27.310427 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.310431 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.310435 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.310439 | controller |     - '        '
2026-03-14 09:44:27.310443 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.310448 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.310452 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.310456 | controller |     - '        '
2026-03-14 09:44:27.310460 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.310464 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.310469 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.310473 | controller |       available"'
2026-03-14 09:44:27.310477 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.310481 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.310485 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.310490 | controller |     - '          exit 1'
2026-03-14 09:44:27.310494 | controller |     - '        fi'
2026-03-14 09:44:27.310498 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.310502 | controller |     - '        '
2026-03-14 09:44:27.310507 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.310511 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.310515 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.310519 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.310523 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.310527 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.310533 | controller |       -- \'
2026-03-14 09:44:27.310540 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.310550 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.310555 | controller |     - '    logger.go:42: 09:30:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.310559 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.310564 | controller |     - '    logger.go:42: 09:30:32 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.310568 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.310572 | controller |       found'
2026-03-14 09:44:27.310576 | controller |     - '    logger.go:42: 09:30:33 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.310582 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.310587 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.310591 | controller |     - '        '
2026-03-14 09:44:27.310595 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.310599 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.310603 | controller |     - '        '
2026-03-14 09:44:27.310608 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.310612 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.310618 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.310624 | controller |     - '          exit 1'
2026-03-14 09:44:27.310630 | controller |     - '        fi'
2026-03-14 09:44:27.310636 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.310643 | controller |     - '        '
2026-03-14 09:44:27.310650 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.310654 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.310659 | controller |       | base64 -d)'
2026-03-14 09:44:27.310663 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.310667 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.310671 | controller |     - '          exit 1'
2026-03-14 09:44:27.310676 | controller |     - '        fi'
2026-03-14 09:44:27.310680 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.310684 | controller |     - '        '
2026-03-14 09:44:27.310688 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.310692 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.310697 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.310701 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.310705 | controller |       --timeout=300s'
2026-03-14 09:44:27.310711 | controller |     - '        '
2026-03-14 09:44:27.310715 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.310719 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.310724 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.310728 | controller |     - '        '
2026-03-14 09:44:27.310732 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.310736 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.310740 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.310745 | controller |     - '        '
2026-03-14 09:44:27.310749 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.310753 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.310757 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.310761 | controller |       available"'
2026-03-14 09:44:27.310766 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.310770 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.310774 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.310778 | controller |     - '          exit 1'
2026-03-14 09:44:27.310784 | controller |     - '        fi'
2026-03-14 09:44:27.310788 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.310794 | controller |     - '        '
2026-03-14 09:44:27.310799 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.310803 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.310807 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.310811 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.310815 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.310820 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.310824 | controller |       -- \'
2026-03-14 09:44:27.310829 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.310834 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.310845 | controller |     - '    logger.go:42: 09:30:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.310850 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.310855 | controller |     - '    logger.go:42: 09:30:33 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.310859 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.310863 | controller |       found'
2026-03-14 09:44:27.310868 | controller |     - '    logger.go:42: 09:30:34 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.310872 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.310876 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.310880 | controller |     - '        '
2026-03-14 09:44:27.310885 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.310889 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.310893 | controller |     - '        '
2026-03-14 09:44:27.310897 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.310901 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.310906 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.310910 | controller |     - '          exit 1'
2026-03-14 09:44:27.310914 | controller |     - '        fi'
2026-03-14 09:44:27.310918 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.310922 | controller |     - '        '
2026-03-14 09:44:27.310927 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.310932 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.310937 | controller |       | base64 -d)'
2026-03-14 09:44:27.310942 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.310946 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.310951 | controller |     - '          exit 1'
2026-03-14 09:44:27.310959 | controller |     - '        fi'
2026-03-14 09:44:27.310964 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.310969 | controller |     - '        '
2026-03-14 09:44:27.310975 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.310981 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.310987 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.310993 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.310999 | controller |       --timeout=300s'
2026-03-14 09:44:27.311005 | controller |     - '        '
2026-03-14 09:44:27.311011 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.311019 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.311023 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.311027 | controller |     - '        '
2026-03-14 09:44:27.311032 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.311036 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.311040 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.311044 | controller |     - '        '
2026-03-14 09:44:27.311048 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.311053 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.311057 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.311061 | controller |       available"'
2026-03-14 09:44:27.311065 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.311069 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.311074 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.311078 | controller |     - '          exit 1'
2026-03-14 09:44:27.311082 | controller |     - '        fi'
2026-03-14 09:44:27.311086 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.311090 | controller |     - '        '
2026-03-14 09:44:27.311094 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.311099 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.311103 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.311107 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.311111 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.311115 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.311120 | controller |       -- \'
2026-03-14 09:44:27.311124 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.311128 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.311139 | controller |     - '    logger.go:42: 09:30:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.311143 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.311148 | controller |     - '    logger.go:42: 09:30:34 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.311152 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.311156 | controller |       found'
2026-03-14 09:44:27.311160 | controller |     - '    logger.go:42: 09:30:35 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.311165 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.311169 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.311173 | controller |     - '        '
2026-03-14 09:44:27.311177 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.311181 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.311186 | controller |     - '        '
2026-03-14 09:44:27.311190 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.311196 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.311201 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.311205 | controller |     - '          exit 1'
2026-03-14 09:44:27.311209 | controller |     - '        fi'
2026-03-14 09:44:27.311213 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.311217 | controller |     - '        '
2026-03-14 09:44:27.311222 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.311226 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.311230 | controller |       | base64 -d)'
2026-03-14 09:44:27.311234 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.311238 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.311243 | controller |     - '          exit 1'
2026-03-14 09:44:27.311247 | controller |     - '        fi'
2026-03-14 09:44:27.311251 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.311255 | controller |     - '        '
2026-03-14 09:44:27.311259 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.311264 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.311268 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.311272 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.311276 | controller |       --timeout=300s'
2026-03-14 09:44:27.311280 | controller |     - '        '
2026-03-14 09:44:27.311284 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.311303 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.311314 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.311318 | controller |     - '        '
2026-03-14 09:44:27.311322 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.311327 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.311331 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.311335 | controller |     - '        '
2026-03-14 09:44:27.311339 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.311344 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.311348 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.311352 | controller |       available"'
2026-03-14 09:44:27.311356 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.311362 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.311366 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.311370 | controller |     - '          exit 1'
2026-03-14 09:44:27.311375 | controller |     - '        fi'
2026-03-14 09:44:27.311379 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.311383 | controller |     - '        '
2026-03-14 09:44:27.311387 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.311391 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.311396 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.311400 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.311406 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.311410 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.311414 | controller |       -- \'
2026-03-14 09:44:27.311418 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.311422 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.311427 | controller |     - '    logger.go:42: 09:30:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.311431 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.311441 | controller |     - '    logger.go:42: 09:30:35 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.311446 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.311450 | controller |       found'
2026-03-14 09:44:27.311454 | controller |     - '    logger.go:42: 09:30:36 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.311458 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.311462 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.311467 | controller |     - '        '
2026-03-14 09:44:27.311471 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.311475 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.311479 | controller |     - '        '
2026-03-14 09:44:27.311483 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.311488 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.311494 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.311498 | controller |     - '          exit 1'
2026-03-14 09:44:27.311502 | controller |     - '        fi'
2026-03-14 09:44:27.311507 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.311511 | controller |     - '        '
2026-03-14 09:44:27.311515 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.311520 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.311524 | controller |       | base64 -d)'
2026-03-14 09:44:27.311528 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.311532 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.311536 | controller |     - '          exit 1'
2026-03-14 09:44:27.311541 | controller |     - '        fi'
2026-03-14 09:44:27.311545 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.311549 | controller |     - '        '
2026-03-14 09:44:27.311553 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.311558 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.311562 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.311566 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.311570 | controller |       --timeout=300s'
2026-03-14 09:44:27.311574 | controller |     - '        '
2026-03-14 09:44:27.311579 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.311583 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.311587 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.311591 | controller |     - '        '
2026-03-14 09:44:27.311596 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.311600 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.311606 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.311610 | controller |     - '        '
2026-03-14 09:44:27.311616 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.311622 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.311628 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.311634 | controller |       available"'
2026-03-14 09:44:27.311640 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.311645 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.311650 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.311654 | controller |     - '          exit 1'
2026-03-14 09:44:27.311658 | controller |     - '        fi'
2026-03-14 09:44:27.311662 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.311667 | controller |     - '        '
2026-03-14 09:44:27.311671 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.311675 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.311679 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.311684 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.311688 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.311692 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.311696 | controller |       -- \'
2026-03-14 09:44:27.311701 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.311705 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.311709 | controller |     - '    logger.go:42: 09:30:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.311716 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.311726 | controller |     - '    logger.go:42: 09:30:37 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.311731 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.311735 | controller |       found'
2026-03-14 09:44:27.311740 | controller |     - '    logger.go:42: 09:30:38 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.311744 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.311748 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.311753 | controller |     - '        '
2026-03-14 09:44:27.311757 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.311761 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.311765 | controller |     - '        '
2026-03-14 09:44:27.311770 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.311774 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.311778 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.311782 | controller |     - '          exit 1'
2026-03-14 09:44:27.311786 | controller |     - '        fi'
2026-03-14 09:44:27.311791 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.311795 | controller |     - '        '
2026-03-14 09:44:27.311799 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.311803 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.311809 | controller |       | base64 -d)'
2026-03-14 09:44:27.311814 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.311818 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.311822 | controller |     - '          exit 1'
2026-03-14 09:44:27.311826 | controller |     - '        fi'
2026-03-14 09:44:27.311830 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.311835 | controller |     - '        '
2026-03-14 09:44:27.311839 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.311843 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.311847 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.311851 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.311856 | controller |       --timeout=300s'
2026-03-14 09:44:27.311860 | controller |     - '        '
2026-03-14 09:44:27.311864 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.311868 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.311872 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.311877 | controller |     - '        '
2026-03-14 09:44:27.311881 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.311885 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.311889 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.311893 | controller |     - '        '
2026-03-14 09:44:27.311897 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.311902 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.311906 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.311910 | controller |       available"'
2026-03-14 09:44:27.311914 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.311920 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.311924 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.311929 | controller |     - '          exit 1'
2026-03-14 09:44:27.311933 | controller |     - '        fi'
2026-03-14 09:44:27.311937 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.311941 | controller |     - '        '
2026-03-14 09:44:27.311947 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.311951 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.311956 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.311961 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.311966 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.311970 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.311975 | controller |       -- \'
2026-03-14 09:44:27.311980 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.311985 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.311991 | controller |     - '    logger.go:42: 09:30:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.311995 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.312000 | controller |     - '    logger.go:42: 09:30:38 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.312009 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.312014 | controller |       found'
2026-03-14 09:44:27.312018 | controller |     - '    logger.go:42: 09:30:39 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.312022 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.312026 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.312031 | controller |     - '        '
2026-03-14 09:44:27.312035 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.312039 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.312044 | controller |     - '        '
2026-03-14 09:44:27.312052 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.312058 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.312064 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.312070 | controller |     - '          exit 1'
2026-03-14 09:44:27.312076 | controller |     - '        fi'
2026-03-14 09:44:27.312082 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.312088 | controller |     - '        '
2026-03-14 09:44:27.312093 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.312097 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.312101 | controller |       | base64 -d)'
2026-03-14 09:44:27.312105 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.312110 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.312114 | controller |     - '          exit 1'
2026-03-14 09:44:27.312118 | controller |     - '        fi'
2026-03-14 09:44:27.312122 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.312127 | controller |     - '        '
2026-03-14 09:44:27.312131 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.312135 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.312139 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.312143 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.312148 | controller |       --timeout=300s'
2026-03-14 09:44:27.312152 | controller |     - '        '
2026-03-14 09:44:27.312156 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.312160 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.312164 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.312169 | controller |     - '        '
2026-03-14 09:44:27.312173 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.312177 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.312181 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.312185 | controller |     - '        '
2026-03-14 09:44:27.312190 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.312194 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.312198 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.312205 | controller |       available"'
2026-03-14 09:44:27.312209 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.312213 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.312217 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.312222 | controller |     - '          exit 1'
2026-03-14 09:44:27.312226 | controller |     - '        fi'
2026-03-14 09:44:27.312230 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.312234 | controller |     - '        '
2026-03-14 09:44:27.312238 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.312243 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.312247 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.312251 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.312258 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.312262 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.312266 | controller |       -- \'
2026-03-14 09:44:27.312270 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.312275 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.312279 | controller |     - '    logger.go:42: 09:30:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.312283 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.312300 | controller |     - '    logger.go:42: 09:30:39 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.312314 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.312319 | controller |       found'
2026-03-14 09:44:27.312323 | controller |     - '    logger.go:42: 09:30:40 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.312327 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.312331 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.312336 | controller |     - '        '
2026-03-14 09:44:27.312340 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.312344 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.312348 | controller |     - '        '
2026-03-14 09:44:27.312353 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.312358 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.312362 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.312366 | controller |     - '          exit 1'
2026-03-14 09:44:27.312370 | controller |     - '        fi'
2026-03-14 09:44:27.312375 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.312379 | controller |     - '        '
2026-03-14 09:44:27.312383 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.312387 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.312392 | controller |       | base64 -d)'
2026-03-14 09:44:27.312396 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.312400 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.312404 | controller |     - '          exit 1'
2026-03-14 09:44:27.312408 | controller |     - '        fi'
2026-03-14 09:44:27.312413 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.312417 | controller |     - '        '
2026-03-14 09:44:27.312421 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.312429 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.312433 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.312437 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.312442 | controller |       --timeout=300s'
2026-03-14 09:44:27.312446 | controller |     - '        '
2026-03-14 09:44:27.312450 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.312454 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.312459 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.312463 | controller |     - '        '
2026-03-14 09:44:27.312467 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.312471 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.312475 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.312480 | controller |     - '        '
2026-03-14 09:44:27.312484 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.312488 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.312492 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.312497 | controller |       available"'
2026-03-14 09:44:27.312501 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.312505 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.312509 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.312513 | controller |     - '          exit 1'
2026-03-14 09:44:27.312518 | controller |     - '        fi'
2026-03-14 09:44:27.312522 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.312526 | controller |     - '        '
2026-03-14 09:44:27.312530 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.312535 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.312539 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.312546 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.312550 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.312554 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.312559 | controller |       -- \'
2026-03-14 09:44:27.312563 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.312567 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.312571 | controller |     - '    logger.go:42: 09:30:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.312576 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.312580 | controller |     - '    logger.go:42: 09:30:40 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.312584 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.312588 | controller |       found'
2026-03-14 09:44:27.312598 | controller |     - '    logger.go:42: 09:30:41 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.312603 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.312609 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.312614 | controller |     - '        '
2026-03-14 09:44:27.312620 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.312626 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.312632 | controller |     - '        '
2026-03-14 09:44:27.312637 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.312643 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.312649 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.312655 | controller |     - '          exit 1'
2026-03-14 09:44:27.312660 | controller |     - '        fi'
2026-03-14 09:44:27.312666 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.312671 | controller |     - '        '
2026-03-14 09:44:27.312677 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.312682 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.312688 | controller |       | base64 -d)'
2026-03-14 09:44:27.312692 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.312696 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.312700 | controller |     - '          exit 1'
2026-03-14 09:44:27.312704 | controller |     - '        fi'
2026-03-14 09:44:27.312709 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.312713 | controller |     - '        '
2026-03-14 09:44:27.312717 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.312721 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.312726 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.312730 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.312734 | controller |       --timeout=300s'
2026-03-14 09:44:27.312738 | controller |     - '        '
2026-03-14 09:44:27.312742 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.312747 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.312751 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.312755 | controller |     - '        '
2026-03-14 09:44:27.312759 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.312763 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.312768 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.312772 | controller |     - '        '
2026-03-14 09:44:27.312776 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.312780 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.312784 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.312789 | controller |       available"'
2026-03-14 09:44:27.312793 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.312797 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.312801 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.312805 | controller |     - '          exit 1'
2026-03-14 09:44:27.312810 | controller |     - '        fi'
2026-03-14 09:44:27.312814 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.312821 | controller |     - '        '
2026-03-14 09:44:27.312825 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.312829 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.312833 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.312837 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.312842 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.312846 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.312850 | controller |       -- \'
2026-03-14 09:44:27.312854 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.312859 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.312863 | controller |     - '    logger.go:42: 09:30:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.312867 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.312871 | controller |     - '    logger.go:42: 09:30:41 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.312875 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.312879 | controller |       found'
2026-03-14 09:44:27.312884 | controller |     - '    logger.go:42: 09:30:42 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.312894 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.312899 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.312903 | controller |     - '        '
2026-03-14 09:44:27.312907 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.312912 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.312916 | controller |     - '        '
2026-03-14 09:44:27.312925 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.312932 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.312936 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.312941 | controller |     - '          exit 1'
2026-03-14 09:44:27.312945 | controller |     - '        fi'
2026-03-14 09:44:27.312949 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.312953 | controller |     - '        '
2026-03-14 09:44:27.312958 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.312962 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.312966 | controller |       | base64 -d)'
2026-03-14 09:44:27.312971 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.312975 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.312979 | controller |     - '          exit 1'
2026-03-14 09:44:27.312984 | controller |     - '        fi'
2026-03-14 09:44:27.312989 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.312997 | controller |     - '        '
2026-03-14 09:44:27.313003 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.313008 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.313012 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.313016 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.313024 | controller |       --timeout=300s'
2026-03-14 09:44:27.313045 | controller |     - '        '
2026-03-14 09:44:27.313050 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.313056 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.313060 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.313065 | controller |     - '        '
2026-03-14 09:44:27.313076 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.313087 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.313092 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.313096 | controller |     - '        '
2026-03-14 09:44:27.313101 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.313105 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.313110 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.313114 | controller |       available"'
2026-03-14 09:44:27.313120 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.313126 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.313132 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.313138 | controller |     - '          exit 1'
2026-03-14 09:44:27.313144 | controller |     - '        fi'
2026-03-14 09:44:27.313150 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.313157 | controller |     - '        '
2026-03-14 09:44:27.313162 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.313168 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.313173 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.313179 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.313184 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.313190 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.313196 | controller |       -- \'
2026-03-14 09:44:27.313201 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.313210 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.313215 | controller |     - '    logger.go:42: 09:30:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.313221 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.313227 | controller |     - '    logger.go:42: 09:30:42 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.313233 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.313239 | controller |       found'
2026-03-14 09:44:27.313244 | controller |     - '    logger.go:42: 09:30:43 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.313250 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.313265 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.313272 | controller |     - '        '
2026-03-14 09:44:27.313278 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.313283 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.313309 | controller |     - '        '
2026-03-14 09:44:27.313318 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.313324 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.313334 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.313340 | controller |     - '          exit 1'
2026-03-14 09:44:27.313345 | controller |     - '        fi'
2026-03-14 09:44:27.313351 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.313356 | controller |     - '        '
2026-03-14 09:44:27.313365 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.313371 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.313376 | controller |       | base64 -d)'
2026-03-14 09:44:27.313382 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.313388 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.313393 | controller |     - '          exit 1'
2026-03-14 09:44:27.313399 | controller |     - '        fi'
2026-03-14 09:44:27.313404 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.313410 | controller |     - '        '
2026-03-14 09:44:27.313415 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.313421 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.313427 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.313432 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.313438 | controller |       --timeout=300s'
2026-03-14 09:44:27.313444 | controller |     - '        '
2026-03-14 09:44:27.313450 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.313456 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.313461 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.313465 | controller |     - '        '
2026-03-14 09:44:27.313469 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.313476 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.313480 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.313485 | controller |     - '        '
2026-03-14 09:44:27.313489 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.313493 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.313497 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.313502 | controller |       available"'
2026-03-14 09:44:27.313506 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.313510 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.313514 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.313519 | controller |     - '          exit 1'
2026-03-14 09:44:27.313523 | controller |     - '        fi'
2026-03-14 09:44:27.313527 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.313531 | controller |     - '        '
2026-03-14 09:44:27.313535 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.313540 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.313544 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.313548 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.313552 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.313559 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.313563 | controller |       -- \'
2026-03-14 09:44:27.313567 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.313571 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.313575 | controller |     - '    logger.go:42: 09:30:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.313580 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.313584 | controller |     - '    logger.go:42: 09:30:43 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.313588 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.313593 | controller |       found'
2026-03-14 09:44:27.313597 | controller |     - '    logger.go:42: 09:30:44 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.313601 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.313605 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.313609 | controller |     - '        '
2026-03-14 09:44:27.313624 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.313631 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.313637 | controller |     - '        '
2026-03-14 09:44:27.313643 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.313649 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.313654 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.313658 | controller |     - '          exit 1'
2026-03-14 09:44:27.313662 | controller |     - '        fi'
2026-03-14 09:44:27.313667 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.313671 | controller |     - '        '
2026-03-14 09:44:27.313675 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.313679 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.313683 | controller |       | base64 -d)'
2026-03-14 09:44:27.313688 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.313692 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.313696 | controller |     - '          exit 1'
2026-03-14 09:44:27.313701 | controller |     - '        fi'
2026-03-14 09:44:27.313707 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.313712 | controller |     - '        '
2026-03-14 09:44:27.313718 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.313724 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.313730 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.313736 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.313741 | controller |       --timeout=300s'
2026-03-14 09:44:27.313747 | controller |     - '        '
2026-03-14 09:44:27.313754 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.313759 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.313763 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.313768 | controller |     - '        '
2026-03-14 09:44:27.313772 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.313777 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.313783 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.313788 | controller |     - '        '
2026-03-14 09:44:27.313792 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.313797 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.313802 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.313808 | controller |       available"'
2026-03-14 09:44:27.313814 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.313819 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.313825 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.313831 | controller |     - '          exit 1'
2026-03-14 09:44:27.313837 | controller |     - '        fi'
2026-03-14 09:44:27.313843 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.313849 | controller |     - '        '
2026-03-14 09:44:27.313854 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.313860 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.313866 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.313871 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.313875 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.313880 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.313884 | controller |       -- \'
2026-03-14 09:44:27.313888 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.313892 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.313897 | controller |     - '    logger.go:42: 09:30:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.313901 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.313908 | controller |     - '    logger.go:42: 09:30:44 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.313912 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.313916 | controller |       found'
2026-03-14 09:44:27.313921 | controller |     - '    logger.go:42: 09:30:45 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.313925 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.313929 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.313948 | controller |     - '        '
2026-03-14 09:44:27.313960 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.313965 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.313969 | controller |     - '        '
2026-03-14 09:44:27.313974 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.313978 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.313982 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.313986 | controller |     - '          exit 1'
2026-03-14 09:44:27.313991 | controller |     - '        fi'
2026-03-14 09:44:27.313995 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.313999 | controller |     - '        '
2026-03-14 09:44:27.314003 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.314007 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.314012 | controller |       | base64 -d)'
2026-03-14 09:44:27.314019 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.314024 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.314028 | controller |     - '          exit 1'
2026-03-14 09:44:27.314032 | controller |     - '        fi'
2026-03-14 09:44:27.314036 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.314041 | controller |     - '        '
2026-03-14 09:44:27.314045 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.314049 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.314053 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.314057 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.314062 | controller |       --timeout=300s'
2026-03-14 09:44:27.314066 | controller |     - '        '
2026-03-14 09:44:27.314070 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.314074 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.314078 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.314083 | controller |     - '        '
2026-03-14 09:44:27.314087 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.314091 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.314095 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.314099 | controller |     - '        '
2026-03-14 09:44:27.314104 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.314108 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.314112 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.314116 | controller |       available"'
2026-03-14 09:44:27.314121 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.314125 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.314129 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.314133 | controller |     - '          exit 1'
2026-03-14 09:44:27.314137 | controller |     - '        fi'
2026-03-14 09:44:27.314142 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.314146 | controller |     - '        '
2026-03-14 09:44:27.314150 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.314154 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.314158 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.314162 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.314167 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.314171 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.314175 | controller |       -- \'
2026-03-14 09:44:27.314179 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.314184 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.314188 | controller |     - '    logger.go:42: 09:30:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.314194 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.314198 | controller |     - '    logger.go:42: 09:30:46 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.314203 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.314207 | controller |       found'
2026-03-14 09:44:27.314211 | controller |     - '    logger.go:42: 09:30:47 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.314215 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.314219 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.314224 | controller |     - '        '
2026-03-14 09:44:27.314228 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.314238 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.314242 | controller |     - '        '
2026-03-14 09:44:27.314246 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.314251 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.314255 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.314259 | controller |     - '          exit 1'
2026-03-14 09:44:27.314263 | controller |     - '        fi'
2026-03-14 09:44:27.314267 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.314272 | controller |     - '        '
2026-03-14 09:44:27.314276 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.314280 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.314284 | controller |       | base64 -d)'
2026-03-14 09:44:27.314304 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.314312 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.314316 | controller |     - '          exit 1'
2026-03-14 09:44:27.314321 | controller |     - '        fi'
2026-03-14 09:44:27.314325 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.314329 | controller |     - '        '
2026-03-14 09:44:27.314336 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.314340 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.314344 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.314348 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.314353 | controller |       --timeout=300s'
2026-03-14 09:44:27.314357 | controller |     - '        '
2026-03-14 09:44:27.314361 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.314365 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.314370 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.314374 | controller |     - '        '
2026-03-14 09:44:27.314378 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.314382 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.314387 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.314391 | controller |     - '        '
2026-03-14 09:44:27.314395 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.314399 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.314404 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.314408 | controller |       available"'
2026-03-14 09:44:27.314414 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.314418 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.314422 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.314426 | controller |     - '          exit 1'
2026-03-14 09:44:27.314431 | controller |     - '        fi'
2026-03-14 09:44:27.314435 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.314439 | controller |     - '        '
2026-03-14 09:44:27.314443 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.314448 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.314452 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.314456 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.314460 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.314464 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.314470 | controller |       -- \'
2026-03-14 09:44:27.314475 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.314479 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.314483 | controller |     - '    logger.go:42: 09:30:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.314487 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.314491 | controller |     - '    logger.go:42: 09:30:47 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.314497 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.314501 | controller |       found'
2026-03-14 09:44:27.314505 | controller |     - '    logger.go:42: 09:30:48 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.314510 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.314514 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.314518 | controller |     - '        '
2026-03-14 09:44:27.314522 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.314527 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.314537 | controller |     - '        '
2026-03-14 09:44:27.314541 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.314546 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.314550 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.314554 | controller |     - '          exit 1'
2026-03-14 09:44:27.314558 | controller |     - '        fi'
2026-03-14 09:44:27.314563 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.314567 | controller |     - '        '
2026-03-14 09:44:27.314571 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.314575 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.314580 | controller |       | base64 -d)'
2026-03-14 09:44:27.314584 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.314589 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.314594 | controller |     - '          exit 1'
2026-03-14 09:44:27.314598 | controller |     - '        fi'
2026-03-14 09:44:27.314602 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.314606 | controller |     - '        '
2026-03-14 09:44:27.314611 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.314616 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.314629 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.314635 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.314641 | controller |       --timeout=300s'
2026-03-14 09:44:27.314646 | controller |     - '        '
2026-03-14 09:44:27.314651 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.314655 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.314659 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.314663 | controller |     - '        '
2026-03-14 09:44:27.314668 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.314672 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.314676 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.314680 | controller |     - '        '
2026-03-14 09:44:27.314685 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.314689 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.314693 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.314697 | controller |       available"'
2026-03-14 09:44:27.314701 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.314708 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.314713 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.314717 | controller |     - '          exit 1'
2026-03-14 09:44:27.314721 | controller |     - '        fi'
2026-03-14 09:44:27.314725 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.314729 | controller |     - '        '
2026-03-14 09:44:27.314734 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.314738 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.314742 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.314746 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.314750 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.314755 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.314759 | controller |       -- \'
2026-03-14 09:44:27.314763 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.314767 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.314772 | controller |     - '    logger.go:42: 09:30:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.314776 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.314780 | controller |     - '    logger.go:42: 09:30:48 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.314784 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.314789 | controller |       found'
2026-03-14 09:44:27.314793 | controller |     - '    logger.go:42: 09:30:49 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.314797 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.314801 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.314807 | controller |     - '        '
2026-03-14 09:44:27.314811 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.314816 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.314820 | controller |     - '        '
2026-03-14 09:44:27.314831 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.314836 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.314840 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.314844 | controller |     - '          exit 1'
2026-03-14 09:44:27.314849 | controller |     - '        fi'
2026-03-14 09:44:27.314853 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.314857 | controller |     - '        '
2026-03-14 09:44:27.314861 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.314865 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.314870 | controller |       | base64 -d)'
2026-03-14 09:44:27.314874 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.314878 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.314882 | controller |     - '          exit 1'
2026-03-14 09:44:27.314886 | controller |     - '        fi'
2026-03-14 09:44:27.314891 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.314895 | controller |     - '        '
2026-03-14 09:44:27.314899 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.314903 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.314907 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.314912 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.314933 | controller |       --timeout=300s'
2026-03-14 09:44:27.314940 | controller |     - '        '
2026-03-14 09:44:27.314944 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.314948 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.314953 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.314957 | controller |     - '        '
2026-03-14 09:44:27.314961 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.314965 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.314969 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.314974 | controller |     - '        '
2026-03-14 09:44:27.314978 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.314982 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.314986 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.314990 | controller |       available"'
2026-03-14 09:44:27.314994 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.314999 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.315003 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.315007 | controller |     - '          exit 1'
2026-03-14 09:44:27.315011 | controller |     - '        fi'
2026-03-14 09:44:27.315015 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.315020 | controller |     - '        '
2026-03-14 09:44:27.315024 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.315029 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.315034 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.315038 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.315042 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.315046 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.315051 | controller |       -- \'
2026-03-14 09:44:27.315055 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.315059 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.315063 | controller |     - '    logger.go:42: 09:30:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.315067 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.315072 | controller |     - '    logger.go:42: 09:30:49 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.315076 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.315080 | controller |       found'
2026-03-14 09:44:27.315084 | controller |     - '    logger.go:42: 09:30:50 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.315088 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.315094 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.315098 | controller |     - '        '
2026-03-14 09:44:27.315103 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.315107 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.315111 | controller |     - '        '
2026-03-14 09:44:27.315121 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.315125 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.315130 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.315134 | controller |     - '          exit 1'
2026-03-14 09:44:27.315138 | controller |     - '        fi'
2026-03-14 09:44:27.315142 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.315147 | controller |     - '        '
2026-03-14 09:44:27.315151 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.315155 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.315159 | controller |       | base64 -d)'
2026-03-14 09:44:27.315163 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.315167 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.315172 | controller |     - '          exit 1'
2026-03-14 09:44:27.315176 | controller |     - '        fi'
2026-03-14 09:44:27.315180 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.315184 | controller |     - '        '
2026-03-14 09:44:27.315188 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.315193 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.315197 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.315201 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.315205 | controller |       --timeout=300s'
2026-03-14 09:44:27.315209 | controller |     - '        '
2026-03-14 09:44:27.315214 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.315218 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.315224 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.315228 | controller |     - '        '
2026-03-14 09:44:27.315232 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.315236 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.315241 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.315245 | controller |     - '        '
2026-03-14 09:44:27.315249 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.315253 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.315257 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.315262 | controller |       available"'
2026-03-14 09:44:27.315266 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.315270 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.315274 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.315278 | controller |     - '          exit 1'
2026-03-14 09:44:27.315283 | controller |     - '        fi'
2026-03-14 09:44:27.315289 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.315311 | controller |     - '        '
2026-03-14 09:44:27.315316 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.315320 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.315324 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.315328 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.315332 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.315337 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.315341 | controller |       -- \'
2026-03-14 09:44:27.315345 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.315349 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.315353 | controller |     - '    logger.go:42: 09:30:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.315358 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.315362 | controller |     - '    logger.go:42: 09:30:50 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.315366 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.315370 | controller |       found'
2026-03-14 09:44:27.315374 | controller |     - '    logger.go:42: 09:30:51 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.315379 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.315386 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.315390 | controller |     - '        '
2026-03-14 09:44:27.315395 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.315399 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.315403 | controller |     - '        '
2026-03-14 09:44:27.315408 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.315412 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.315422 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.315430 | controller |     - '          exit 1'
2026-03-14 09:44:27.315434 | controller |     - '        fi'
2026-03-14 09:44:27.315440 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.315445 | controller |     - '        '
2026-03-14 09:44:27.315449 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.315453 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.315457 | controller |       | base64 -d)'
2026-03-14 09:44:27.315462 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.315466 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.315470 | controller |     - '          exit 1'
2026-03-14 09:44:27.315474 | controller |     - '        fi'
2026-03-14 09:44:27.315478 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.315483 | controller |     - '        '
2026-03-14 09:44:27.315487 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.315493 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.315497 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.315501 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.315505 | controller |       --timeout=300s'
2026-03-14 09:44:27.315509 | controller |     - '        '
2026-03-14 09:44:27.315514 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.315518 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.315522 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.315526 | controller |     - '        '
2026-03-14 09:44:27.315530 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.315535 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.315540 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.315544 | controller |     - '        '
2026-03-14 09:44:27.315549 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.315553 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.315557 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.315561 | controller |       available"'
2026-03-14 09:44:27.315566 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.315570 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.315574 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.315578 | controller |     - '          exit 1'
2026-03-14 09:44:27.315582 | controller |     - '        fi'
2026-03-14 09:44:27.315587 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.315591 | controller |     - '        '
2026-03-14 09:44:27.315595 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.315599 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.315603 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.315607 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.315612 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.315621 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.315627 | controller |       -- \'
2026-03-14 09:44:27.315633 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.315639 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.315644 | controller |     - '    logger.go:42: 09:30:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.315650 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.315655 | controller |     - '    logger.go:42: 09:30:51 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.315660 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.315666 | controller |       found'
2026-03-14 09:44:27.315672 | controller |     - '    logger.go:42: 09:30:52 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.315676 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.315680 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.315684 | controller |     - '        '
2026-03-14 09:44:27.315691 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.315695 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.315700 | controller |     - '        '
2026-03-14 09:44:27.315704 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.315708 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.315720 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.315724 | controller |     - '          exit 1'
2026-03-14 09:44:27.315728 | controller |     - '        fi'
2026-03-14 09:44:27.315733 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.315737 | controller |     - '        '
2026-03-14 09:44:27.315741 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.315745 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.315750 | controller |       | base64 -d)'
2026-03-14 09:44:27.315754 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.315758 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.315762 | controller |     - '          exit 1'
2026-03-14 09:44:27.315766 | controller |     - '        fi'
2026-03-14 09:44:27.315771 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.315775 | controller |     - '        '
2026-03-14 09:44:27.315779 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.315783 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.315788 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.315792 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.315796 | controller |       --timeout=300s'
2026-03-14 09:44:27.315800 | controller |     - '        '
2026-03-14 09:44:27.315804 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.315809 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.315813 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.315817 | controller |     - '        '
2026-03-14 09:44:27.315821 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.315825 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.315847 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.315854 | controller |     - '        '
2026-03-14 09:44:27.315858 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.315863 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.315867 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.315871 | controller |       available"'
2026-03-14 09:44:27.315875 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.315880 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.315884 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.315888 | controller |     - '          exit 1'
2026-03-14 09:44:27.315892 | controller |     - '        fi'
2026-03-14 09:44:27.315897 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.315901 | controller |     - '        '
2026-03-14 09:44:27.315905 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.315909 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.315913 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.315918 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.315922 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.315926 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.315930 | controller |       -- \'
2026-03-14 09:44:27.315935 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.315939 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.315945 | controller |     - '    logger.go:42: 09:30:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.315949 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.315953 | controller |     - '    logger.go:42: 09:30:52 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.315958 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.315962 | controller |       found'
2026-03-14 09:44:27.315966 | controller |     - '    logger.go:42: 09:30:53 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.315970 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.315975 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.315979 | controller |     - '        '
2026-03-14 09:44:27.315983 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.315987 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.315992 | controller |     - '        '
2026-03-14 09:44:27.315996 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.316000 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.316004 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.316009 | controller |     - '          exit 1'
2026-03-14 09:44:27.316021 | controller |     - '        fi'
2026-03-14 09:44:27.316028 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.316034 | controller |     - '        '
2026-03-14 09:44:27.316042 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.316048 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.316054 | controller |       | base64 -d)'
2026-03-14 09:44:27.316060 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.316069 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.316074 | controller |     - '          exit 1'
2026-03-14 09:44:27.316081 | controller |     - '        fi'
2026-03-14 09:44:27.316087 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.316093 | controller |     - '        '
2026-03-14 09:44:27.316099 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.316105 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.316111 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.316116 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.316120 | controller |       --timeout=300s'
2026-03-14 09:44:27.316125 | controller |     - '        '
2026-03-14 09:44:27.316129 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.316133 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.316137 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.316141 | controller |     - '        '
2026-03-14 09:44:27.316146 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.316150 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.316154 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.316158 | controller |     - '        '
2026-03-14 09:44:27.316163 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.316167 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.316171 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.316175 | controller |       available"'
2026-03-14 09:44:27.316179 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.316184 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.316188 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.316192 | controller |     - '          exit 1'
2026-03-14 09:44:27.316196 | controller |     - '        fi'
2026-03-14 09:44:27.316200 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.316205 | controller |     - '        '
2026-03-14 09:44:27.316209 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.316213 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.316217 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.316221 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.316226 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.316230 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.316234 | controller |       -- \'
2026-03-14 09:44:27.316238 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.316242 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.316247 | controller |     - '    logger.go:42: 09:30:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.316253 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.316257 | controller |     - '    logger.go:42: 09:30:54 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.316261 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.316266 | controller |       found'
2026-03-14 09:44:27.316270 | controller |     - '    logger.go:42: 09:30:55 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.316274 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.316278 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.316282 | controller |     - '        '
2026-03-14 09:44:27.316301 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.316311 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.316316 | controller |     - '        '
2026-03-14 09:44:27.316320 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.316324 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.316328 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.316333 | controller |     - '          exit 1'
2026-03-14 09:44:27.316337 | controller |     - '        fi'
2026-03-14 09:44:27.316348 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.316353 | controller |     - '        '
2026-03-14 09:44:27.316357 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.316361 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.316365 | controller |       | base64 -d)'
2026-03-14 09:44:27.316369 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.316374 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.316378 | controller |     - '          exit 1'
2026-03-14 09:44:27.316382 | controller |     - '        fi'
2026-03-14 09:44:27.316386 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.316390 | controller |     - '        '
2026-03-14 09:44:27.316395 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.316399 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.316403 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.316407 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.316411 | controller |       --timeout=300s'
2026-03-14 09:44:27.316416 | controller |     - '        '
2026-03-14 09:44:27.316420 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.316424 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.316428 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.316432 | controller |     - '        '
2026-03-14 09:44:27.316436 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.316441 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.316445 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.316449 | controller |     - '        '
2026-03-14 09:44:27.316453 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.316457 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.316461 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.316466 | controller |       available"'
2026-03-14 09:44:27.316470 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.316476 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.316480 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.316484 | controller |     - '          exit 1'
2026-03-14 09:44:27.316489 | controller |     - '        fi'
2026-03-14 09:44:27.316495 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.316499 | controller |     - '        '
2026-03-14 09:44:27.316503 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.316507 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.316511 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.316516 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.316520 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.316524 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.316528 | controller |       -- \'
2026-03-14 09:44:27.316533 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.316537 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.316543 | controller |     - '    logger.go:42: 09:30:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.316547 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.316551 | controller |     - '    logger.go:42: 09:30:55 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.316555 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.316560 | controller |       found'
2026-03-14 09:44:27.316564 | controller |     - '    logger.go:42: 09:30:56 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.316568 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.316572 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.316577 | controller |     - '        '
2026-03-14 09:44:27.316581 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.316585 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.316589 | controller |     - '        '
2026-03-14 09:44:27.316594 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.316599 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.316603 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.316607 | controller |     - '          exit 1'
2026-03-14 09:44:27.316612 | controller |     - '        fi'
2026-03-14 09:44:27.316617 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.316623 | controller |     - '        '
2026-03-14 09:44:27.316638 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.316645 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.316651 | controller |       | base64 -d)'
2026-03-14 09:44:27.316657 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.316662 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.316668 | controller |     - '          exit 1'
2026-03-14 09:44:27.316673 | controller |     - '        fi'
2026-03-14 09:44:27.316679 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.316684 | controller |     - '        '
2026-03-14 09:44:27.316690 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.316695 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.316703 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.316707 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.316711 | controller |       --timeout=300s'
2026-03-14 09:44:27.316715 | controller |     - '        '
2026-03-14 09:44:27.316720 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.316724 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.316728 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.316732 | controller |     - '        '
2026-03-14 09:44:27.316737 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.316741 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.316745 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.316749 | controller |     - '        '
2026-03-14 09:44:27.316754 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.316813 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.316821 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.316826 | controller |       available"'
2026-03-14 09:44:27.316831 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.316835 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.316839 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.316843 | controller |     - '          exit 1'
2026-03-14 09:44:27.316849 | controller |     - '        fi'
2026-03-14 09:44:27.316853 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.316857 | controller |     - '        '
2026-03-14 09:44:27.316862 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.316866 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.316870 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.316874 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.316879 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.316883 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.316887 | controller |       -- \'
2026-03-14 09:44:27.316892 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.316900 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.316916 | controller |     - '    logger.go:42: 09:30:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.316921 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.316926 | controller |     - '    logger.go:42: 09:30:56 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.316930 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.316936 | controller |       found'
2026-03-14 09:44:27.316941 | controller |     - '    logger.go:42: 09:30:57 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.316946 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.316951 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.316955 | controller |     - '        '
2026-03-14 09:44:27.316962 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.316966 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.316970 | controller |     - '        '
2026-03-14 09:44:27.316978 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.316982 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.316986 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.316991 | controller |     - '          exit 1'
2026-03-14 09:44:27.316995 | controller |     - '        fi'
2026-03-14 09:44:27.316999 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.317003 | controller |     - '        '
2026-03-14 09:44:27.317008 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.317018 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.317023 | controller |       | base64 -d)'
2026-03-14 09:44:27.317028 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.317033 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.317037 | controller |     - '          exit 1'
2026-03-14 09:44:27.317041 | controller |     - '        fi'
2026-03-14 09:44:27.317045 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.317050 | controller |     - '        '
2026-03-14 09:44:27.317054 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.317058 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.317062 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.317067 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.317071 | controller |       --timeout=300s'
2026-03-14 09:44:27.317075 | controller |     - '        '
2026-03-14 09:44:27.317080 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.317084 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.317088 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.317093 | controller |     - '        '
2026-03-14 09:44:27.317097 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.317102 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.317106 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.317110 | controller |     - '        '
2026-03-14 09:44:27.317114 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.317119 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.317123 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.317127 | controller |       available"'
2026-03-14 09:44:27.317132 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.317136 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.317140 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.317144 | controller |     - '          exit 1'
2026-03-14 09:44:27.317149 | controller |     - '        fi'
2026-03-14 09:44:27.317153 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.317157 | controller |     - '        '
2026-03-14 09:44:27.317161 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.317167 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.317172 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.317176 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.317180 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.317185 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.317189 | controller |       -- \'
2026-03-14 09:44:27.317193 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.317197 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.317202 | controller |     - '    logger.go:42: 09:30:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.317206 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.317210 | controller |     - '    logger.go:42: 09:30:57 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.317215 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.317219 | controller |       found'
2026-03-14 09:44:27.317223 | controller |     - '    logger.go:42: 09:30:58 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.317227 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.317232 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.317236 | controller |     - '        '
2026-03-14 09:44:27.317240 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.317244 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.317249 | controller |     - '        '
2026-03-14 09:44:27.317253 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.317257 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.317262 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.317266 | controller |     - '          exit 1'
2026-03-14 09:44:27.317270 | controller |     - '        fi'
2026-03-14 09:44:27.317275 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.317279 | controller |     - '        '
2026-03-14 09:44:27.317283 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.317307 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.317315 | controller |       | base64 -d)'
2026-03-14 09:44:27.317320 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.317325 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.317329 | controller |     - '          exit 1'
2026-03-14 09:44:27.317333 | controller |     - '        fi'
2026-03-14 09:44:27.317338 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.317342 | controller |     - '        '
2026-03-14 09:44:27.317346 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.317351 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.317355 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.317359 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.317363 | controller |       --timeout=300s'
2026-03-14 09:44:27.317368 | controller |     - '        '
2026-03-14 09:44:27.317372 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.317376 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.317407 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.317413 | controller |     - '        '
2026-03-14 09:44:27.317420 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.317424 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.317429 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.317433 | controller |     - '        '
2026-03-14 09:44:27.317437 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.317442 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.317446 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.317450 | controller |       available"'
2026-03-14 09:44:27.317454 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.317460 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.317464 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.317469 | controller |     - '          exit 1'
2026-03-14 09:44:27.317473 | controller |     - '        fi'
2026-03-14 09:44:27.317515 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.317523 | controller |     - '        '
2026-03-14 09:44:27.317535 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.317539 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.317543 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.317548 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.317552 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.317556 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.317561 | controller |       -- \'
2026-03-14 09:44:27.317565 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.317569 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.317574 | controller |     - '    logger.go:42: 09:30:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.317582 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.317595 | controller |     - '    logger.go:42: 09:30:58 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.317600 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.317607 | controller |       found'
2026-03-14 09:44:27.317619 | controller |     - '    logger.go:42: 09:30:59 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.317626 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.317632 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.317638 | controller |     - '        '
2026-03-14 09:44:27.317645 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.317653 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.317666 | controller |     - '        '
2026-03-14 09:44:27.317671 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.317677 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.317688 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.317693 | controller |     - '          exit 1'
2026-03-14 09:44:27.317701 | controller |     - '        fi'
2026-03-14 09:44:27.317709 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.317715 | controller |     - '        '
2026-03-14 09:44:27.317720 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.317724 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.317734 | controller |       | base64 -d)'
2026-03-14 09:44:27.317739 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.317745 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.317753 | controller |     - '          exit 1'
2026-03-14 09:44:27.317759 | controller |     - '        fi'
2026-03-14 09:44:27.317796 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.317803 | controller |     - '        '
2026-03-14 09:44:27.317809 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.317815 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.317820 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.317826 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.317831 | controller |       --timeout=300s'
2026-03-14 09:44:27.317838 | controller |     - '        '
2026-03-14 09:44:27.317873 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.317904 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.317914 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.317920 | controller |     - '        '
2026-03-14 09:44:27.317966 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.317975 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.317981 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.317987 | controller |     - '        '
2026-03-14 09:44:27.318007 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.318018 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.318024 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.318030 | controller |       available"'
2026-03-14 09:44:27.318059 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.318068 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.318074 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.318080 | controller |     - '          exit 1'
2026-03-14 09:44:27.318086 | controller |     - '        fi'
2026-03-14 09:44:27.318092 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.318098 | controller |     - '        '
2026-03-14 09:44:27.318103 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.318109 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.318115 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.318121 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.318127 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.318133 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.318167 | controller |       -- \'
2026-03-14 09:44:27.318173 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.318178 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.318183 | controller |     - '    logger.go:42: 09:30:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.318189 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.318195 | controller |     - '    logger.go:42: 09:30:59 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.318201 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.318207 | controller |       found'
2026-03-14 09:44:27.318213 | controller |     - '    logger.go:42: 09:31:00 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.318219 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.318224 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.318230 | controller |     - '        '
2026-03-14 09:44:27.318236 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.318242 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.318247 | controller |     - '        '
2026-03-14 09:44:27.318253 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.318259 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.318265 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.318271 | controller |     - '          exit 1'
2026-03-14 09:44:27.318276 | controller |     - '        fi'
2026-03-14 09:44:27.318282 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.318288 | controller |     - '        '
2026-03-14 09:44:27.318313 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.318319 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.318325 | controller |       | base64 -d)'
2026-03-14 09:44:27.318339 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.318347 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.318352 | controller |     - '          exit 1'
2026-03-14 09:44:27.318358 | controller |     - '        fi'
2026-03-14 09:44:27.318364 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.318370 | controller |     - '        '
2026-03-14 09:44:27.318376 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.318382 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.318388 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.318394 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.318400 | controller |       --timeout=300s'
2026-03-14 09:44:27.318406 | controller |     - '        '
2026-03-14 09:44:27.318412 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.318417 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.318423 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.318428 | controller |     - '        '
2026-03-14 09:44:27.318434 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.318440 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.318445 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.318455 | controller |     - '        '
2026-03-14 09:44:27.318461 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.318467 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.318473 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.318482 | controller |       available"'
2026-03-14 09:44:27.318488 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.318494 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.318500 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.318506 | controller |     - '          exit 1'
2026-03-14 09:44:27.318511 | controller |     - '        fi'
2026-03-14 09:44:27.318517 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.318523 | controller |     - '        '
2026-03-14 09:44:27.318527 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.318531 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.318536 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.318540 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.318544 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.318548 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.318553 | controller |       -- \'
2026-03-14 09:44:27.318557 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.318561 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.318565 | controller |     - '    logger.go:42: 09:31:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.318569 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.318607 | controller |     - '    logger.go:42: 09:31:00 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.318613 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.318619 | controller |       found'
2026-03-14 09:44:27.318624 | controller |     - '    logger.go:42: 09:31:01 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.318631 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.318637 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.318643 | controller |     - '        '
2026-03-14 09:44:27.318649 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.318654 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.318660 | controller |     - '        '
2026-03-14 09:44:27.318666 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.318672 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.318677 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.318683 | controller |     - '          exit 1'
2026-03-14 09:44:27.318689 | controller |     - '        fi'
2026-03-14 09:44:27.318695 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.318701 | controller |     - '        '
2026-03-14 09:44:27.318706 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.318712 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.318718 | controller |       | base64 -d)'
2026-03-14 09:44:27.318724 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.318739 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.318749 | controller |     - '          exit 1'
2026-03-14 09:44:27.318754 | controller |     - '        fi'
2026-03-14 09:44:27.318760 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.318766 | controller |     - '        '
2026-03-14 09:44:27.318772 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.318778 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.318788 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.318792 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.318796 | controller |       --timeout=300s'
2026-03-14 09:44:27.318800 | controller |     - '        '
2026-03-14 09:44:27.318805 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.318809 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.318813 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.318817 | controller |     - '        '
2026-03-14 09:44:27.318822 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.318826 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.318830 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.318834 | controller |     - '        '
2026-03-14 09:44:27.318872 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.318881 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.318887 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.318893 | controller |       available"'
2026-03-14 09:44:27.318898 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.318904 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.318910 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.318916 | controller |     - '          exit 1'
2026-03-14 09:44:27.318921 | controller |     - '        fi'
2026-03-14 09:44:27.318927 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.318933 | controller |     - '        '
2026-03-14 09:44:27.318939 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.318945 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.318950 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.318956 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.318962 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.318968 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.318974 | controller |       -- \'
2026-03-14 09:44:27.318979 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.318985 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.318991 | controller |     - '    logger.go:42: 09:31:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.318997 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.319003 | controller |     - '    logger.go:42: 09:31:01 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.319011 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.319017 | controller |       found'
2026-03-14 09:44:27.319023 | controller |     - '    logger.go:42: 09:31:02 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.319029 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.319063 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.319073 | controller |     - '        '
2026-03-14 09:44:27.319077 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.319084 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.319088 | controller |     - '        '
2026-03-14 09:44:27.319093 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.319097 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.319101 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.319106 | controller |     - '          exit 1'
2026-03-14 09:44:27.319111 | controller |     - '        fi'
2026-03-14 09:44:27.319117 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.319123 | controller |     - '        '
2026-03-14 09:44:27.319143 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.319150 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.319156 | controller |       | base64 -d)'
2026-03-14 09:44:27.319162 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.319177 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.319182 | controller |     - '          exit 1'
2026-03-14 09:44:27.319187 | controller |     - '        fi'
2026-03-14 09:44:27.319191 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.319196 | controller |     - '        '
2026-03-14 09:44:27.319204 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.319218 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.319223 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.319227 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.319232 | controller |       --timeout=300s'
2026-03-14 09:44:27.319237 | controller |     - '        '
2026-03-14 09:44:27.319243 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.319250 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.319254 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.319261 | controller |     - '        '
2026-03-14 09:44:27.319266 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.319271 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.319275 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.319280 | controller |     - '        '
2026-03-14 09:44:27.319286 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.319316 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.319322 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.319326 | controller |       available"'
2026-03-14 09:44:27.319331 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.319335 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.319343 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.319353 | controller |     - '          exit 1'
2026-03-14 09:44:27.319357 | controller |     - '        fi'
2026-03-14 09:44:27.319361 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.319369 | controller |     - '        '
2026-03-14 09:44:27.319381 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.319387 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.319394 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.319399 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.319409 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.319420 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.319424 | controller |       -- \'
2026-03-14 09:44:27.319428 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.319433 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.319437 | controller |     - '    logger.go:42: 09:31:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.319441 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.319445 | controller |     - '    logger.go:42: 09:31:03 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.319450 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.319454 | controller |       found'
2026-03-14 09:44:27.319458 | controller |     - '    logger.go:42: 09:31:04 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.319462 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.319466 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.319471 | controller |     - '        '
2026-03-14 09:44:27.319475 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.319479 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.319483 | controller |     - '        '
2026-03-14 09:44:27.319488 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.319492 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.319496 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.319500 | controller |     - '          exit 1'
2026-03-14 09:44:27.319504 | controller |     - '        fi'
2026-03-14 09:44:27.319509 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.319513 | controller |     - '        '
2026-03-14 09:44:27.319520 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.319524 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.319529 | controller |       | base64 -d)'
2026-03-14 09:44:27.319534 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.319539 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.319544 | controller |     - '          exit 1'
2026-03-14 09:44:27.319548 | controller |     - '        fi'
2026-03-14 09:44:27.319560 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.319564 | controller |     - '        '
2026-03-14 09:44:27.319569 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.319573 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.319577 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.319585 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.319589 | controller |       --timeout=300s'
2026-03-14 09:44:27.319593 | controller |     - '        '
2026-03-14 09:44:27.319598 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.319602 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.319607 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.319612 | controller |     - '        '
2026-03-14 09:44:27.319617 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.319623 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.319628 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.319634 | controller |     - '        '
2026-03-14 09:44:27.319641 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.319646 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.319652 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.319658 | controller |       available"'
2026-03-14 09:44:27.319663 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.319669 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.319675 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.319680 | controller |     - '          exit 1'
2026-03-14 09:44:27.319685 | controller |     - '        fi'
2026-03-14 09:44:27.319691 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.319697 | controller |     - '        '
2026-03-14 09:44:27.319702 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.319707 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.319711 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.319715 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.319719 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.319723 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.319728 | controller |       -- \'
2026-03-14 09:44:27.319732 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.319736 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.319740 | controller |     - '    logger.go:42: 09:31:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.319744 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.319749 | controller |     - '    logger.go:42: 09:31:04 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.319753 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.319757 | controller |       found'
2026-03-14 09:44:27.319761 | controller |     - '    logger.go:42: 09:31:05 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.319765 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.319770 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.319774 | controller |     - '        '
2026-03-14 09:44:27.319778 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.319784 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.319789 | controller |     - '        '
2026-03-14 09:44:27.319793 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.319839 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.319851 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.319864 | controller |     - '          exit 1'
2026-03-14 09:44:27.319869 | controller |     - '        fi'
2026-03-14 09:44:27.319873 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.319877 | controller |     - '        '
2026-03-14 09:44:27.319882 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.319886 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.319890 | controller |       | base64 -d)'
2026-03-14 09:44:27.319896 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.319900 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.319904 | controller |     - '          exit 1'
2026-03-14 09:44:27.319909 | controller |     - '        fi'
2026-03-14 09:44:27.319920 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.319928 | controller |     - '        '
2026-03-14 09:44:27.319933 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.319939 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.319945 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.319949 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.319957 | controller |       --timeout=300s'
2026-03-14 09:44:27.319961 | controller |     - '        '
2026-03-14 09:44:27.319965 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.319970 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.319975 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.319985 | controller |     - '        '
2026-03-14 09:44:27.319989 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.319993 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.319998 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.320002 | controller |     - '        '
2026-03-14 09:44:27.320006 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.320010 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.320015 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.320019 | controller |       available"'
2026-03-14 09:44:27.320023 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.320028 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.320032 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.320039 | controller |     - '          exit 1'
2026-03-14 09:44:27.320053 | controller |     - '        fi'
2026-03-14 09:44:27.320057 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.320061 | controller |     - '        '
2026-03-14 09:44:27.320066 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.320071 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.320077 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.320086 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.320091 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.320095 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.320100 | controller |       -- \'
2026-03-14 09:44:27.320105 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.320112 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.320123 | controller |     - '    logger.go:42: 09:31:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.320128 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.320132 | controller |     - '    logger.go:42: 09:31:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.320137 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.320144 | controller |       found'
2026-03-14 09:44:27.320156 | controller |     - '    logger.go:42: 09:31:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.320161 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.320168 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.320173 | controller |     - '        '
2026-03-14 09:44:27.320177 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.320182 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.320189 | controller |     - '        '
2026-03-14 09:44:27.320202 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.320206 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.320210 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.320215 | controller |     - '          exit 1'
2026-03-14 09:44:27.320219 | controller |     - '        fi'
2026-03-14 09:44:27.320224 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.320231 | controller |     - '        '
2026-03-14 09:44:27.320243 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.320247 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.320252 | controller |       | base64 -d)'
2026-03-14 09:44:27.320259 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.320273 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.320278 | controller |     - '          exit 1'
2026-03-14 09:44:27.320282 | controller |     - '        fi'
2026-03-14 09:44:27.320289 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.320313 | controller |     - '        '
2026-03-14 09:44:27.320373 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.320382 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.320386 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.320390 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.320395 | controller |       --timeout=300s'
2026-03-14 09:44:27.320402 | controller |     - '        '
2026-03-14 09:44:27.320408 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.320415 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.320424 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.320434 | controller |     - '        '
2026-03-14 09:44:27.320440 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.320446 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.320452 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.320461 | controller |     - '        '
2026-03-14 09:44:27.320486 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.320494 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.320500 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.320509 | controller |       available"'
2026-03-14 09:44:27.320515 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.320524 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.320534 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.320539 | controller |     - '          exit 1'
2026-03-14 09:44:27.320543 | controller |     - '        fi'
2026-03-14 09:44:27.320548 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.320555 | controller |     - '        '
2026-03-14 09:44:27.320566 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.320570 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.320574 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.320581 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.320592 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.320596 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.320603 | controller |       -- \'
2026-03-14 09:44:27.320628 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.320638 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.320687 | controller |     - '    logger.go:42: 09:31:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.320693 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.320702 | controller |     - '    logger.go:42: 09:31:06 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.320715 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.320720 | controller |       found'
2026-03-14 09:44:27.320725 | controller |     - '    logger.go:42: 09:31:07 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.320729 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.320733 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.320738 | controller |     - '        '
2026-03-14 09:44:27.320742 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.320746 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.320751 | controller |     - '        '
2026-03-14 09:44:27.320755 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.320759 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.320764 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.320768 | controller |     - '          exit 1'
2026-03-14 09:44:27.320772 | controller |     - '        fi'
2026-03-14 09:44:27.320780 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.320785 | controller |     - '        '
2026-03-14 09:44:27.320789 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.320793 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.320798 | controller |       | base64 -d)'
2026-03-14 09:44:27.320802 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.320836 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.320842 | controller |     - '          exit 1'
2026-03-14 09:44:27.320847 | controller |     - '        fi'
2026-03-14 09:44:27.320852 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.320856 | controller |     - '        '
2026-03-14 09:44:27.320860 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.320872 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.320876 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.320881 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.320885 | controller |       --timeout=300s'
2026-03-14 09:44:27.320890 | controller |     - '        '
2026-03-14 09:44:27.320894 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.320898 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.320903 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.320907 | controller |     - '        '
2026-03-14 09:44:27.320911 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.320915 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.320920 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.320924 | controller |     - '        '
2026-03-14 09:44:27.320928 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.320932 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.320937 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.320941 | controller |       available"'
2026-03-14 09:44:27.320945 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.320949 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.320954 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.320958 | controller |     - '          exit 1'
2026-03-14 09:44:27.320962 | controller |     - '        fi'
2026-03-14 09:44:27.320967 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.320971 | controller |     - '        '
2026-03-14 09:44:27.320975 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.320979 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.320984 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.320988 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.320996 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.321000 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.321006 | controller |       -- \'
2026-03-14 09:44:27.321010 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.321015 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.321020 | controller |     - '    logger.go:42: 09:31:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.321024 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.321029 | controller |     - '    logger.go:42: 09:31:07 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.321033 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.321037 | controller |       found'
2026-03-14 09:44:27.321041 | controller |     - '    logger.go:42: 09:31:08 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.321046 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.321050 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.321054 | controller |     - '        '
2026-03-14 09:44:27.321061 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.321065 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.321069 | controller |     - '        '
2026-03-14 09:44:27.321074 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.321078 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.321082 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.321086 | controller |     - '          exit 1'
2026-03-14 09:44:27.321091 | controller |     - '        fi'
2026-03-14 09:44:27.321095 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.321099 | controller |     - '        '
2026-03-14 09:44:27.321104 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.321108 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.321112 | controller |       | base64 -d)'
2026-03-14 09:44:27.321117 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.321122 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.321126 | controller |     - '          exit 1'
2026-03-14 09:44:27.321130 | controller |     - '        fi'
2026-03-14 09:44:27.321134 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.321139 | controller |     - '        '
2026-03-14 09:44:27.321143 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.321153 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.321157 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.321162 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.321166 | controller |       --timeout=300s'
2026-03-14 09:44:27.321170 | controller |     - '        '
2026-03-14 09:44:27.321175 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.321179 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.321183 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.321187 | controller |     - '        '
2026-03-14 09:44:27.321192 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.321196 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.321200 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.321204 | controller |     - '        '
2026-03-14 09:44:27.321210 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.321215 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.321221 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.321226 | controller |       available"'
2026-03-14 09:44:27.321230 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.321236 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.321240 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.321244 | controller |     - '          exit 1'
2026-03-14 09:44:27.321249 | controller |     - '        fi'
2026-03-14 09:44:27.321253 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.321257 | controller |     - '        '
2026-03-14 09:44:27.321261 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.321266 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.321270 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.321274 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.321279 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.321283 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.321306 | controller |       -- \'
2026-03-14 09:44:27.321314 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.321319 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.321326 | controller |     - '    logger.go:42: 09:31:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.321330 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.321334 | controller |     - '    logger.go:42: 09:31:08 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.321339 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.321343 | controller |       found'
2026-03-14 09:44:27.321347 | controller |     - '    logger.go:42: 09:31:09 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.321351 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.321356 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.321360 | controller |     - '        '
2026-03-14 09:44:27.321364 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.321369 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.321373 | controller |     - '        '
2026-03-14 09:44:27.321377 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.321382 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.321386 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.321390 | controller |     - '          exit 1'
2026-03-14 09:44:27.321394 | controller |     - '        fi'
2026-03-14 09:44:27.321399 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.321403 | controller |     - '        '
2026-03-14 09:44:27.321407 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.321412 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.321416 | controller |       | base64 -d)'
2026-03-14 09:44:27.321420 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.321424 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.321430 | controller |     - '          exit 1'
2026-03-14 09:44:27.321435 | controller |     - '        fi'
2026-03-14 09:44:27.321439 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.321443 | controller |     - '        '
2026-03-14 09:44:27.321448 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.321452 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.321463 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.321493 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.321498 | controller |       --timeout=300s'
2026-03-14 09:44:27.321503 | controller |     - '        '
2026-03-14 09:44:27.321507 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.321511 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.321516 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.321520 | controller |     - '        '
2026-03-14 09:44:27.321524 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.321529 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.321533 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.321537 | controller |     - '        '
2026-03-14 09:44:27.321542 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.321546 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.321550 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.321555 | controller |       available"'
2026-03-14 09:44:27.321559 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.321563 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.321567 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.321572 | controller |     - '          exit 1'
2026-03-14 09:44:27.321576 | controller |     - '        fi'
2026-03-14 09:44:27.321580 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.321584 | controller |     - '        '
2026-03-14 09:44:27.321589 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.321593 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.321597 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.321601 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.321605 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.321610 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.321614 | controller |       -- \'
2026-03-14 09:44:27.321620 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.321628 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.321635 | controller |     - '    logger.go:42: 09:31:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.321641 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.321647 | controller |     - '    logger.go:42: 09:31:09 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.321658 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.321662 | controller |       found'
2026-03-14 09:44:27.321667 | controller |     - '    logger.go:42: 09:31:10 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.321671 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.321675 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.321679 | controller |     - '        '
2026-03-14 09:44:27.321684 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.321688 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.321692 | controller |     - '        '
2026-03-14 09:44:27.321696 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.321700 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.321705 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.321709 | controller |     - '          exit 1'
2026-03-14 09:44:27.321713 | controller |     - '        fi'
2026-03-14 09:44:27.321717 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.321722 | controller |     - '        '
2026-03-14 09:44:27.321726 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.321730 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.321734 | controller |       | base64 -d)'
2026-03-14 09:44:27.321738 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.321743 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.321747 | controller |     - '          exit 1'
2026-03-14 09:44:27.321751 | controller |     - '        fi'
2026-03-14 09:44:27.321755 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.321759 | controller |     - '        '
2026-03-14 09:44:27.321764 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.321768 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.321772 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.321784 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.321789 | controller |       --timeout=300s'
2026-03-14 09:44:27.321793 | controller |     - '        '
2026-03-14 09:44:27.321797 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.321801 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.321806 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.321810 | controller |     - '        '
2026-03-14 09:44:27.321814 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.321818 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.321822 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.321827 | controller |     - '        '
2026-03-14 09:44:27.321831 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.321835 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.321839 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.321844 | controller |       available"'
2026-03-14 09:44:27.321848 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.321854 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.321860 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.321864 | controller |     - '          exit 1'
2026-03-14 09:44:27.321869 | controller |     - '        fi'
2026-03-14 09:44:27.321873 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.321877 | controller |     - '        '
2026-03-14 09:44:27.321881 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.321885 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.321890 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.321894 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.321898 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.321902 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.321906 | controller |       -- \'
2026-03-14 09:44:27.321911 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.321915 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.321919 | controller |     - '    logger.go:42: 09:31:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.321923 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.321927 | controller |     - '    logger.go:42: 09:31:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.321932 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.321936 | controller |       found'
2026-03-14 09:44:27.321940 | controller |     - '    logger.go:42: 09:31:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.321944 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.321989 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.321994 | controller |     - '        '
2026-03-14 09:44:27.321998 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.322003 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.322007 | controller |     - '        '
2026-03-14 09:44:27.322011 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.322015 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.322019 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.322024 | controller |     - '          exit 1'
2026-03-14 09:44:27.322028 | controller |     - '        fi'
2026-03-14 09:44:27.322032 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.322036 | controller |     - '        '
2026-03-14 09:44:27.322041 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.322045 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.322049 | controller |       | base64 -d)'
2026-03-14 09:44:27.322053 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.322057 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.322062 | controller |     - '          exit 1'
2026-03-14 09:44:27.322066 | controller |     - '        fi'
2026-03-14 09:44:27.322070 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.322098 | controller |     - '        '
2026-03-14 09:44:27.322104 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.322109 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.322113 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.322126 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.322131 | controller |       --timeout=300s'
2026-03-14 09:44:27.322135 | controller |     - '        '
2026-03-14 09:44:27.322139 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.322144 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.322148 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.322152 | controller |     - '        '
2026-03-14 09:44:27.322156 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.322163 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.322167 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.322171 | controller |     - '        '
2026-03-14 09:44:27.322178 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.322183 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.322187 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.322191 | controller |       available"'
2026-03-14 09:44:27.322195 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.322200 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.322204 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.322208 | controller |     - '          exit 1'
2026-03-14 09:44:27.322212 | controller |     - '        fi'
2026-03-14 09:44:27.322217 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.322221 | controller |     - '        '
2026-03-14 09:44:27.322225 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.322229 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.322234 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.322238 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.322242 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.322246 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.322250 | controller |       -- \'
2026-03-14 09:44:27.322255 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.322259 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.322263 | controller |     - '    logger.go:42: 09:31:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.322267 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.322271 | controller |     - '    logger.go:42: 09:31:12 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.322276 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.322280 | controller |       found'
2026-03-14 09:44:27.322284 | controller |     - '    logger.go:42: 09:31:13 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.322300 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.322308 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.322312 | controller |     - '        '
2026-03-14 09:44:27.322316 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.322320 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.322327 | controller |     - '        '
2026-03-14 09:44:27.322331 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.322335 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.322339 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.322344 | controller |     - '          exit 1'
2026-03-14 09:44:27.322348 | controller |     - '        fi'
2026-03-14 09:44:27.322352 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.322356 | controller |     - '        '
2026-03-14 09:44:27.322360 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.322365 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.322369 | controller |       | base64 -d)'
2026-03-14 09:44:27.322373 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.322377 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.322382 | controller |     - '          exit 1'
2026-03-14 09:44:27.322386 | controller |     - '        fi'
2026-03-14 09:44:27.322390 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.322394 | controller |     - '        '
2026-03-14 09:44:27.322398 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.322403 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.322407 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.322411 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.322415 | controller |       --timeout=300s'
2026-03-14 09:44:27.322426 | controller |     - '        '
2026-03-14 09:44:27.322431 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.322435 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.322439 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.322444 | controller |     - '        '
2026-03-14 09:44:27.322448 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.322452 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.322456 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.322461 | controller |     - '        '
2026-03-14 09:44:27.322465 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.322469 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.322473 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.322477 | controller |       available"'
2026-03-14 09:44:27.322482 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.322486 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.322490 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.322494 | controller |     - '          exit 1'
2026-03-14 09:44:27.322499 | controller |     - '        fi'
2026-03-14 09:44:27.322503 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.322507 | controller |     - '        '
2026-03-14 09:44:27.322513 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.322517 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.322523 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.322527 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.322532 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.322536 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.322540 | controller |       -- \'
2026-03-14 09:44:27.322544 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.322548 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.322553 | controller |     - '    logger.go:42: 09:31:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.322557 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.322561 | controller |     - '    logger.go:42: 09:31:13 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.322565 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.322569 | controller |       found'
2026-03-14 09:44:27.322574 | controller |     - '    logger.go:42: 09:31:14 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.322578 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.322582 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.322586 | controller |     - '        '
2026-03-14 09:44:27.322590 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.322595 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.322599 | controller |     - '        '
2026-03-14 09:44:27.322603 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.322607 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.322611 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.322616 | controller |     - '          exit 1'
2026-03-14 09:44:27.322622 | controller |     - '        fi'
2026-03-14 09:44:27.322628 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.322634 | controller |     - '        '
2026-03-14 09:44:27.322658 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.322664 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.322668 | controller |       | base64 -d)'
2026-03-14 09:44:27.322673 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.322677 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.322681 | controller |     - '          exit 1'
2026-03-14 09:44:27.322685 | controller |     - '        fi'
2026-03-14 09:44:27.322690 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.322694 | controller |     - '        '
2026-03-14 09:44:27.322698 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.322702 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.322706 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.322711 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.322715 | controller |       --timeout=300s'
2026-03-14 09:44:27.322719 | controller |     - '        '
2026-03-14 09:44:27.322731 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.322736 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.322740 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.322747 | controller |     - '        '
2026-03-14 09:44:27.322753 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.322758 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.322762 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.322766 | controller |     - '        '
2026-03-14 09:44:27.322770 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.322774 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.322779 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.322783 | controller |       available"'
2026-03-14 09:44:27.322787 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.322791 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.322795 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.322799 | controller |     - '          exit 1'
2026-03-14 09:44:27.322804 | controller |     - '        fi'
2026-03-14 09:44:27.322808 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.322812 | controller |     - '        '
2026-03-14 09:44:27.322816 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.322821 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.322825 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.322829 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.322833 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.322837 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.322841 | controller |       -- \'
2026-03-14 09:44:27.322846 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.322850 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.322854 | controller |     - '    logger.go:42: 09:31:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.322858 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.322863 | controller |     - '    logger.go:42: 09:31:14 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.322867 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.322871 | controller |       found'
2026-03-14 09:44:27.322875 | controller |     - '    logger.go:42: 09:31:15 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.322879 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.322884 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.322888 | controller |     - '        '
2026-03-14 09:44:27.322892 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.322896 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.322900 | controller |     - '        '
2026-03-14 09:44:27.322905 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.322909 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.322913 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.322917 | controller |     - '          exit 1'
2026-03-14 09:44:27.322922 | controller |     - '        fi'
2026-03-14 09:44:27.322926 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.322932 | controller |     - '        '
2026-03-14 09:44:27.322937 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.322943 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.322947 | controller |       | base64 -d)'
2026-03-14 09:44:27.322952 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.322957 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.322962 | controller |     - '          exit 1'
2026-03-14 09:44:27.322967 | controller |     - '        fi'
2026-03-14 09:44:27.322971 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.322975 | controller |     - '        '
2026-03-14 09:44:27.322979 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.322983 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.322988 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.322992 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.322996 | controller |       --timeout=300s'
2026-03-14 09:44:27.323000 | controller |     - '        '
2026-03-14 09:44:27.323011 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.323015 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.323019 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.323024 | controller |     - '        '
2026-03-14 09:44:27.323028 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.323032 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.323036 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.323041 | controller |     - '        '
2026-03-14 09:44:27.323045 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.323049 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.323053 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.323057 | controller |       available"'
2026-03-14 09:44:27.323062 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.323066 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.323070 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.323074 | controller |     - '          exit 1'
2026-03-14 09:44:27.323079 | controller |     - '        fi'
2026-03-14 09:44:27.323083 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.323087 | controller |     - '        '
2026-03-14 09:44:27.323091 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.323095 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.323100 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.323106 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.323110 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.323114 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.323120 | controller |       -- \'
2026-03-14 09:44:27.323124 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.323130 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.323134 | controller |     - '    logger.go:42: 09:31:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.323139 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.323143 | controller |     - '    logger.go:42: 09:31:15 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.323147 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.323151 | controller |       found'
2026-03-14 09:44:27.323156 | controller |     - '    logger.go:42: 09:31:16 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.323160 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.323164 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.323168 | controller |     - '        '
2026-03-14 09:44:27.323173 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.323177 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.323181 | controller |     - '        '
2026-03-14 09:44:27.323185 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.323189 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.323194 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.323198 | controller |     - '          exit 1'
2026-03-14 09:44:27.323202 | controller |     - '        fi'
2026-03-14 09:44:27.323206 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.323211 | controller |     - '        '
2026-03-14 09:44:27.323215 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.323219 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.323223 | controller |       | base64 -d)'
2026-03-14 09:44:27.323227 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.323232 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.323236 | controller |     - '          exit 1'
2026-03-14 09:44:27.323240 | controller |     - '        fi'
2026-03-14 09:44:27.323246 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.323250 | controller |     - '        '
2026-03-14 09:44:27.323255 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.323259 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.323263 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.323267 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.323271 | controller |       --timeout=300s'
2026-03-14 09:44:27.323276 | controller |     - '        '
2026-03-14 09:44:27.323280 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.323301 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.323308 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.323313 | controller |     - '        '
2026-03-14 09:44:27.323317 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.323321 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.323325 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.323330 | controller |     - '        '
2026-03-14 09:44:27.323334 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.323340 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.323344 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.323348 | controller |       available"'
2026-03-14 09:44:27.323353 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.323357 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.323361 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.323365 | controller |     - '          exit 1'
2026-03-14 09:44:27.323369 | controller |     - '        fi'
2026-03-14 09:44:27.323374 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.323378 | controller |     - '        '
2026-03-14 09:44:27.323382 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.323386 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.323391 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.323395 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.323399 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.323403 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.323407 | controller |       -- \'
2026-03-14 09:44:27.323412 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.323416 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.323420 | controller |     - '    logger.go:42: 09:31:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.323424 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.323428 | controller |     - '    logger.go:42: 09:31:16 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.323433 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.323437 | controller |       found'
2026-03-14 09:44:27.323441 | controller |     - '    logger.go:42: 09:31:17 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.323445 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.323449 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.323454 | controller |     - '        '
2026-03-14 09:44:27.323458 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.323462 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.323466 | controller |     - '        '
2026-03-14 09:44:27.323471 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.323475 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.323479 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.323483 | controller |     - '          exit 1'
2026-03-14 09:44:27.323487 | controller |     - '        fi'
2026-03-14 09:44:27.323492 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.323496 | controller |     - '        '
2026-03-14 09:44:27.323500 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.323504 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.323508 | controller |       | base64 -d)'
2026-03-14 09:44:27.323513 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.323517 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.323521 | controller |     - '          exit 1'
2026-03-14 09:44:27.323527 | controller |     - '        fi'
2026-03-14 09:44:27.323531 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.323535 | controller |     - '        '
2026-03-14 09:44:27.323539 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.323544 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.323549 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.323555 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.323560 | controller |       --timeout=300s'
2026-03-14 09:44:27.323566 | controller |     - '        '
2026-03-14 09:44:27.323572 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.323583 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.323589 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.323595 | controller |     - '        '
2026-03-14 09:44:27.323601 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.323606 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.323612 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.323617 | controller |     - '        '
2026-03-14 09:44:27.323623 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.323629 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.323635 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.323641 | controller |       available"'
2026-03-14 09:44:27.323647 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.323653 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.323659 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.323665 | controller |     - '          exit 1'
2026-03-14 09:44:27.323671 | controller |     - '        fi'
2026-03-14 09:44:27.323676 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.323681 | controller |     - '        '
2026-03-14 09:44:27.323685 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.323689 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.323693 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.323697 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.323702 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.323706 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.323710 | controller |       -- \'
2026-03-14 09:44:27.323714 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.323722 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.323726 | controller |     - '    logger.go:42: 09:31:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.323731 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.323735 | controller |     - '    logger.go:42: 09:31:17 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.323740 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.323747 | controller |       found'
2026-03-14 09:44:27.323751 | controller |     - '    logger.go:42: 09:31:18 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.323756 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.323760 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.323764 | controller |     - '        '
2026-03-14 09:44:27.323768 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.323772 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.323777 | controller |     - '        '
2026-03-14 09:44:27.323781 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.323785 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.323789 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.323794 | controller |     - '          exit 1'
2026-03-14 09:44:27.323798 | controller |     - '        fi'
2026-03-14 09:44:27.323802 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.323806 | controller |     - '        '
2026-03-14 09:44:27.323810 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.323814 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.323819 | controller |       | base64 -d)'
2026-03-14 09:44:27.323823 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.323827 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.323831 | controller |     - '          exit 1'
2026-03-14 09:44:27.323835 | controller |     - '        fi'
2026-03-14 09:44:27.323840 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.323844 | controller |     - '        '
2026-03-14 09:44:27.323848 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.323852 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.323857 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.323861 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.323865 | controller |       --timeout=300s'
2026-03-14 09:44:27.323869 | controller |     - '        '
2026-03-14 09:44:27.323873 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.323878 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.323890 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.323894 | controller |     - '        '
2026-03-14 09:44:27.323899 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.323903 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.323908 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.323912 | controller |     - '        '
2026-03-14 09:44:27.323916 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.323920 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.323924 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.323929 | controller |       available"'
2026-03-14 09:44:27.323933 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.323939 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.323943 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.323949 | controller |     - '          exit 1'
2026-03-14 09:44:27.323953 | controller |     - '        fi'
2026-03-14 09:44:27.323957 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.323961 | controller |     - '        '
2026-03-14 09:44:27.323966 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.323970 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.323976 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.323982 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.323987 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.323993 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.323999 | controller |       -- \'
2026-03-14 09:44:27.324004 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.324010 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.324016 | controller |     - '    logger.go:42: 09:31:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.324021 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.324027 | controller |     - '    logger.go:42: 09:31:18 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.324033 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.324039 | controller |       found'
2026-03-14 09:44:27.324045 | controller |     - '    logger.go:42: 09:31:19 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.324051 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.324060 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.324067 | controller |     - '        '
2026-03-14 09:44:27.324073 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.324078 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.324084 | controller |     - '        '
2026-03-14 09:44:27.324090 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.324096 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.324101 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.324107 | controller |     - '          exit 1'
2026-03-14 09:44:27.324113 | controller |     - '        fi'
2026-03-14 09:44:27.324119 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.324124 | controller |     - '        '
2026-03-14 09:44:27.324130 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.324136 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.324142 | controller |       | base64 -d)'
2026-03-14 09:44:27.324148 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.324155 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.324159 | controller |     - '          exit 1'
2026-03-14 09:44:27.324163 | controller |     - '        fi'
2026-03-14 09:44:27.324168 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.324172 | controller |     - '        '
2026-03-14 09:44:27.324176 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.324180 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.324184 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.324188 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.324195 | controller |       --timeout=300s'
2026-03-14 09:44:27.324199 | controller |     - '        '
2026-03-14 09:44:27.324203 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.324208 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.324221 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.324226 | controller |     - '        '
2026-03-14 09:44:27.324230 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.324234 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.324240 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.324244 | controller |     - '        '
2026-03-14 09:44:27.324249 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.324253 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.324257 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.324261 | controller |       available"'
2026-03-14 09:44:27.324265 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.324270 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.324274 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.324278 | controller |     - '          exit 1'
2026-03-14 09:44:27.324282 | controller |     - '        fi'
2026-03-14 09:44:27.324289 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.324309 | controller |     - '        '
2026-03-14 09:44:27.324314 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.324319 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.324323 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.324327 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.324332 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.324338 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.324344 | controller |       -- \'
2026-03-14 09:44:27.324349 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.324355 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.324361 | controller |     - '    logger.go:42: 09:31:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.324367 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.324374 | controller |     - '    logger.go:42: 09:31:20 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.324378 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.324382 | controller |       found'
2026-03-14 09:44:27.324386 | controller |     - '    logger.go:42: 09:31:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.324390 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.324395 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.324399 | controller |     - '        '
2026-03-14 09:44:27.324403 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.324407 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.324413 | controller |     - '        '
2026-03-14 09:44:27.324418 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.324422 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.324426 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.324430 | controller |     - '          exit 1'
2026-03-14 09:44:27.324436 | controller |     - '        fi'
2026-03-14 09:44:27.324440 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.324445 | controller |     - '        '
2026-03-14 09:44:27.324449 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.324453 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.324457 | controller |       | base64 -d)'
2026-03-14 09:44:27.324461 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.324466 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.324470 | controller |     - '          exit 1'
2026-03-14 09:44:27.324474 | controller |     - '        fi'
2026-03-14 09:44:27.324478 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.324482 | controller |     - '        '
2026-03-14 09:44:27.324487 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.324491 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.324495 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.324499 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.324503 | controller |       --timeout=300s'
2026-03-14 09:44:27.324509 | controller |     - '        '
2026-03-14 09:44:27.324513 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.324518 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.324522 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.324533 | controller |     - '        '
2026-03-14 09:44:27.324538 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.324542 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.324546 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.324550 | controller |     - '        '
2026-03-14 09:44:27.324554 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.324559 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.324563 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.324567 | controller |       available"'
2026-03-14 09:44:27.324571 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.324575 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.324580 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.324584 | controller |     - '          exit 1'
2026-03-14 09:44:27.324588 | controller |     - '        fi'
2026-03-14 09:44:27.324592 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.324597 | controller |     - '        '
2026-03-14 09:44:27.324601 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.324605 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.324609 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.324616 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.324622 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.324627 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.324633 | controller |       -- \'
2026-03-14 09:44:27.324639 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.324645 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.324651 | controller |     - '    logger.go:42: 09:31:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.324656 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.324660 | controller |     - '    logger.go:42: 09:31:21 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.324664 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.324668 | controller |       found'
2026-03-14 09:44:27.324673 | controller |     - '    logger.go:42: 09:31:22 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.324677 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.324681 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.324685 | controller |     - '        '
2026-03-14 09:44:27.324689 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.324694 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.324698 | controller |     - '        '
2026-03-14 09:44:27.324702 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.324706 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.324710 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.324714 | controller |     - '          exit 1'
2026-03-14 09:44:27.324719 | controller |     - '        fi'
2026-03-14 09:44:27.324723 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.324727 | controller |     - '        '
2026-03-14 09:44:27.324731 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.324735 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.324739 | controller |       | base64 -d)'
2026-03-14 09:44:27.324744 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.324748 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.324752 | controller |     - '          exit 1'
2026-03-14 09:44:27.324756 | controller |     - '        fi'
2026-03-14 09:44:27.324760 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.324765 | controller |     - '        '
2026-03-14 09:44:27.324769 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.324776 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.324781 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.324785 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.324789 | controller |       --timeout=300s'
2026-03-14 09:44:27.324794 | controller |     - '        '
2026-03-14 09:44:27.324798 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.324802 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.324806 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.324810 | controller |     - '        '
2026-03-14 09:44:27.324823 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.324827 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.324832 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.324836 | controller |     - '        '
2026-03-14 09:44:27.324840 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.324844 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.324848 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.324852 | controller |       available"'
2026-03-14 09:44:27.324857 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.324861 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.324865 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.324869 | controller |     - '          exit 1'
2026-03-14 09:44:27.324874 | controller |     - '        fi'
2026-03-14 09:44:27.324878 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.324882 | controller |     - '        '
2026-03-14 09:44:27.324886 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.324890 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.324894 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.324899 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.324903 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.324907 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.324911 | controller |       -- \'
2026-03-14 09:44:27.324915 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.324919 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.324924 | controller |     - '    logger.go:42: 09:31:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.324928 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.324932 | controller |     - '    logger.go:42: 09:31:22 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.324938 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.324943 | controller |       found'
2026-03-14 09:44:27.324948 | controller |     - '    logger.go:42: 09:31:23 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.324952 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.324956 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.324962 | controller |     - '        '
2026-03-14 09:44:27.324967 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.324971 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.324976 | controller |     - '        '
2026-03-14 09:44:27.324981 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.324986 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.324991 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.324995 | controller |     - '          exit 1'
2026-03-14 09:44:27.324999 | controller |     - '        fi'
2026-03-14 09:44:27.325003 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.325008 | controller |     - '        '
2026-03-14 09:44:27.325015 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.325019 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.325023 | controller |       | base64 -d)'
2026-03-14 09:44:27.325027 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.325032 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.325036 | controller |     - '          exit 1'
2026-03-14 09:44:27.325040 | controller |     - '        fi'
2026-03-14 09:44:27.325044 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.325049 | controller |     - '        '
2026-03-14 09:44:27.325053 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.325057 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.325061 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.325065 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.325069 | controller |       --timeout=300s'
2026-03-14 09:44:27.325074 | controller |     - '        '
2026-03-14 09:44:27.325078 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.325082 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.325086 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.325090 | controller |     - '        '
2026-03-14 09:44:27.325100 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.325105 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.325109 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.325113 | controller |     - '        '
2026-03-14 09:44:27.325118 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.325122 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.325126 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.325130 | controller |       available"'
2026-03-14 09:44:27.325134 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.325140 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.325145 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.325151 | controller |     - '          exit 1'
2026-03-14 09:44:27.325157 | controller |     - '        fi'
2026-03-14 09:44:27.325163 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.325168 | controller |     - '        '
2026-03-14 09:44:27.325174 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.325180 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.325186 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.325192 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.325196 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.325200 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.325204 | controller |       -- \'
2026-03-14 09:44:27.325209 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.325215 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.325221 | controller |     - '    logger.go:42: 09:31:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.325226 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.325230 | controller |     - '    logger.go:42: 09:31:23 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.325234 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.325238 | controller |       found'
2026-03-14 09:44:27.325243 | controller |     - '    logger.go:42: 09:31:24 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.325247 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.325251 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.325255 | controller |     - '        '
2026-03-14 09:44:27.325259 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.325264 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.325268 | controller |     - '        '
2026-03-14 09:44:27.325272 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.325276 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.325280 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.325285 | controller |     - '          exit 1'
2026-03-14 09:44:27.325317 | controller |     - '        fi'
2026-03-14 09:44:27.325322 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.325326 | controller |     - '        '
2026-03-14 09:44:27.325330 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.325334 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.325339 | controller |       | base64 -d)'
2026-03-14 09:44:27.325343 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.325347 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.325354 | controller |     - '          exit 1'
2026-03-14 09:44:27.325358 | controller |     - '        fi'
2026-03-14 09:44:27.325362 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.325367 | controller |     - '        '
2026-03-14 09:44:27.325371 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.325375 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.325379 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.325383 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.325387 | controller |       --timeout=300s'
2026-03-14 09:44:27.325392 | controller |     - '        '
2026-03-14 09:44:27.325396 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.325400 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.325404 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.325409 | controller |     - '        '
2026-03-14 09:44:27.325413 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.325424 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.325429 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.325433 | controller |     - '        '
2026-03-14 09:44:27.325437 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.325443 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.325448 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.325452 | controller |       available"'
2026-03-14 09:44:27.325456 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.325460 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.325465 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.325469 | controller |     - '          exit 1'
2026-03-14 09:44:27.325475 | controller |     - '        fi'
2026-03-14 09:44:27.325479 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.325483 | controller |     - '        '
2026-03-14 09:44:27.325487 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.325492 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.325496 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.325500 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.325504 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.325508 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.325513 | controller |       -- \'
2026-03-14 09:44:27.325517 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.325521 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.325527 | controller |     - '    logger.go:42: 09:31:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.325531 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.325535 | controller |     - '    logger.go:42: 09:31:24 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.325539 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.325544 | controller |       found'
2026-03-14 09:44:27.325549 | controller |     - '    logger.go:42: 09:31:25 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.325554 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.325558 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.325562 | controller |     - '        '
2026-03-14 09:44:27.325566 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.325571 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.325575 | controller |     - '        '
2026-03-14 09:44:27.325579 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.325583 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.325587 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.325592 | controller |     - '          exit 1'
2026-03-14 09:44:27.325596 | controller |     - '        fi'
2026-03-14 09:44:27.325600 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.325604 | controller |     - '        '
2026-03-14 09:44:27.325608 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.325613 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.325618 | controller |       | base64 -d)'
2026-03-14 09:44:27.325624 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.325630 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.325636 | controller |     - '          exit 1'
2026-03-14 09:44:27.325642 | controller |     - '        fi'
2026-03-14 09:44:27.325650 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.325654 | controller |     - '        '
2026-03-14 09:44:27.325658 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.325662 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.325667 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.325671 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.325675 | controller |       --timeout=300s'
2026-03-14 09:44:27.325679 | controller |     - '        '
2026-03-14 09:44:27.325683 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.325688 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.325692 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.325696 | controller |     - '        '
2026-03-14 09:44:27.325700 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.325711 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.325716 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.325720 | controller |     - '        '
2026-03-14 09:44:27.325725 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.325729 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.325733 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.325737 | controller |       available"'
2026-03-14 09:44:27.325741 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.325746 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.325750 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.325754 | controller |     - '          exit 1'
2026-03-14 09:44:27.325758 | controller |     - '        fi'
2026-03-14 09:44:27.325763 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.325767 | controller |     - '        '
2026-03-14 09:44:27.325771 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.325775 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.325779 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.325783 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.325788 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.325792 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.325796 | controller |       -- \'
2026-03-14 09:44:27.325800 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.325805 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.325809 | controller |     - '    logger.go:42: 09:31:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.325813 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.325817 | controller |     - '    logger.go:42: 09:31:25 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.325822 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.325827 | controller |       found'
2026-03-14 09:44:27.325832 | controller |     - '    logger.go:42: 09:31:26 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.325836 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.325840 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.325844 | controller |     - '        '
2026-03-14 09:44:27.325848 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.325852 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.325857 | controller |     - '        '
2026-03-14 09:44:27.325861 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.325865 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.325869 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.325873 | controller |     - '          exit 1'
2026-03-14 09:44:27.325878 | controller |     - '        fi'
2026-03-14 09:44:27.325882 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.325886 | controller |     - '        '
2026-03-14 09:44:27.325890 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.325894 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.325899 | controller |       | base64 -d)'
2026-03-14 09:44:27.325903 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.325907 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.325911 | controller |     - '          exit 1'
2026-03-14 09:44:27.325915 | controller |     - '        fi'
2026-03-14 09:44:27.325919 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.325924 | controller |     - '        '
2026-03-14 09:44:27.325928 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.325932 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.325936 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.325940 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.325945 | controller |       --timeout=300s'
2026-03-14 09:44:27.325949 | controller |     - '        '
2026-03-14 09:44:27.325953 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.326013 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.326018 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.326022 | controller |     - '        '
2026-03-14 09:44:27.326027 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.326031 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.326041 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.326045 | controller |     - '        '
2026-03-14 09:44:27.326050 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.326054 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.326058 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.326063 | controller |       available"'
2026-03-14 09:44:27.326068 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.326074 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.326080 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.326086 | controller |     - '          exit 1'
2026-03-14 09:44:27.326094 | controller |     - '        fi'
2026-03-14 09:44:27.326100 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.326104 | controller |     - '        '
2026-03-14 09:44:27.326109 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.326113 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.326117 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.326121 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.326127 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.326132 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.326138 | controller |       -- \'
2026-03-14 09:44:27.326144 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.326150 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.326156 | controller |     - '    logger.go:42: 09:31:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.326162 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.326168 | controller |     - '    logger.go:42: 09:31:27 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.326173 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.326179 | controller |       found'
2026-03-14 09:44:27.326185 | controller |     - '    logger.go:42: 09:31:28 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.326191 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.326197 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.326204 | controller |     - '        '
2026-03-14 09:44:27.326214 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.326220 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.326226 | controller |     - '        '
2026-03-14 09:44:27.326232 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.326238 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.326244 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.326250 | controller |     - '          exit 1'
2026-03-14 09:44:27.326256 | controller |     - '        fi'
2026-03-14 09:44:27.326262 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.326268 | controller |     - '        '
2026-03-14 09:44:27.326274 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.326278 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.326282 | controller |       | base64 -d)'
2026-03-14 09:44:27.326304 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.326316 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.326322 | controller |     - '          exit 1'
2026-03-14 09:44:27.326328 | controller |     - '        fi'
2026-03-14 09:44:27.326334 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.326339 | controller |     - '        '
2026-03-14 09:44:27.326344 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.326349 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.326353 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.326357 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.326361 | controller |       --timeout=300s'
2026-03-14 09:44:27.326368 | controller |     - '        '
2026-03-14 09:44:27.326373 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.326377 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.326381 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.326385 | controller |     - '        '
2026-03-14 09:44:27.326389 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.326394 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.326406 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.326410 | controller |     - '        '
2026-03-14 09:44:27.326415 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.326424 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.326429 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.326435 | controller |       available"'
2026-03-14 09:44:27.326441 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.326447 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.326453 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.326459 | controller |     - '          exit 1'
2026-03-14 09:44:27.326464 | controller |     - '        fi'
2026-03-14 09:44:27.326470 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.326476 | controller |     - '        '
2026-03-14 09:44:27.326482 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.326488 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.326493 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.326498 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.326502 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.326506 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.326510 | controller |       -- \'
2026-03-14 09:44:27.326515 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.326519 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.326523 | controller |     - '    logger.go:42: 09:31:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.326527 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.326531 | controller |     - '    logger.go:42: 09:31:28 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.326536 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.326540 | controller |       found'
2026-03-14 09:44:27.326544 | controller |     - '    logger.go:42: 09:31:29 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.326548 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.326552 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.326557 | controller |     - '        '
2026-03-14 09:44:27.326561 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.326565 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.326569 | controller |     - '        '
2026-03-14 09:44:27.326575 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.326580 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.326584 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.326588 | controller |     - '          exit 1'
2026-03-14 09:44:27.326592 | controller |     - '        fi'
2026-03-14 09:44:27.326597 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.326601 | controller |     - '        '
2026-03-14 09:44:27.326605 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.326609 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.326613 | controller |       | base64 -d)'
2026-03-14 09:44:27.326619 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.326625 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.326631 | controller |     - '          exit 1'
2026-03-14 09:44:27.326637 | controller |     - '        fi'
2026-03-14 09:44:27.326643 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.326648 | controller |     - '        '
2026-03-14 09:44:27.326654 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.326660 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.326666 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.326676 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.326682 | controller |       --timeout=300s'
2026-03-14 09:44:27.326690 | controller |     - '        '
2026-03-14 09:44:27.326695 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.326701 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.326707 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.326713 | controller |     - '        '
2026-03-14 09:44:27.326719 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.326724 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.326729 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.326733 | controller |     - '        '
2026-03-14 09:44:27.326744 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.326749 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.326754 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.326758 | controller |       available"'
2026-03-14 09:44:27.326762 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.326768 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.326772 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.326777 | controller |     - '          exit 1'
2026-03-14 09:44:27.326781 | controller |     - '        fi'
2026-03-14 09:44:27.326785 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.326789 | controller |     - '        '
2026-03-14 09:44:27.326794 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.326798 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.326802 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.326808 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.326812 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.326817 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.326821 | controller |       -- \'
2026-03-14 09:44:27.326825 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.326829 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.326833 | controller |     - '    logger.go:42: 09:31:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.326838 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.326842 | controller |     - '    logger.go:42: 09:31:29 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.326846 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.326850 | controller |       found'
2026-03-14 09:44:27.326855 | controller |     - '    logger.go:42: 09:31:30 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.326859 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.326863 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.326868 | controller |     - '        '
2026-03-14 09:44:27.326872 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.326878 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.326882 | controller |     - '        '
2026-03-14 09:44:27.326886 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.326891 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.326895 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.326899 | controller |     - '          exit 1'
2026-03-14 09:44:27.326903 | controller |     - '        fi'
2026-03-14 09:44:27.326907 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.326912 | controller |     - '        '
2026-03-14 09:44:27.326917 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.326922 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.326927 | controller |       | base64 -d)'
2026-03-14 09:44:27.326932 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.326937 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.326942 | controller |     - '          exit 1'
2026-03-14 09:44:27.326946 | controller |     - '        fi'
2026-03-14 09:44:27.326950 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.326955 | controller |     - '        '
2026-03-14 09:44:27.326959 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.326963 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.326967 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.326971 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.326976 | controller |       --timeout=300s'
2026-03-14 09:44:27.326980 | controller |     - '        '
2026-03-14 09:44:27.326984 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.326988 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.326993 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.326997 | controller |     - '        '
2026-03-14 09:44:27.327002 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.327009 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.327013 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.327017 | controller |     - '        '
2026-03-14 09:44:27.327026 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.327033 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.327038 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.327044 | controller |       available"'
2026-03-14 09:44:27.327050 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.327056 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.327061 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.327067 | controller |     - '          exit 1'
2026-03-14 09:44:27.327072 | controller |     - '        fi'
2026-03-14 09:44:27.327076 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.327081 | controller |     - '        '
2026-03-14 09:44:27.327085 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.327089 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.327093 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.327097 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.327102 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.327106 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.327110 | controller |       -- \'
2026-03-14 09:44:27.327114 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.327118 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.327123 | controller |     - '    logger.go:42: 09:31:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.327127 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.327131 | controller |     - '    logger.go:42: 09:31:30 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.327135 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.327139 | controller |       found'
2026-03-14 09:44:27.327144 | controller |     - '    logger.go:42: 09:31:31 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.327148 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.327152 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.327156 | controller |     - '        '
2026-03-14 09:44:27.327160 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.327164 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.327169 | controller |     - '        '
2026-03-14 09:44:27.327173 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.327177 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.327181 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.327185 | controller |     - '          exit 1'
2026-03-14 09:44:27.327190 | controller |     - '        fi'
2026-03-14 09:44:27.327194 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.327198 | controller |     - '        '
2026-03-14 09:44:27.327202 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.327208 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.327212 | controller |       | base64 -d)'
2026-03-14 09:44:27.327217 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.327221 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.327225 | controller |     - '          exit 1'
2026-03-14 09:44:27.327229 | controller |     - '        fi'
2026-03-14 09:44:27.327233 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.327238 | controller |     - '        '
2026-03-14 09:44:27.327242 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.327246 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.327250 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.327254 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.327259 | controller |       --timeout=300s'
2026-03-14 09:44:27.327263 | controller |     - '        '
2026-03-14 09:44:27.327267 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.327271 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.327275 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.327280 | controller |     - '        '
2026-03-14 09:44:27.327284 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.327302 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.327309 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.327314 | controller |     - '        '
2026-03-14 09:44:27.327318 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.327328 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.327333 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.327337 | controller |       available"'
2026-03-14 09:44:27.327341 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.327345 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.327349 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.327354 | controller |     - '          exit 1'
2026-03-14 09:44:27.327358 | controller |     - '        fi'
2026-03-14 09:44:27.327362 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.327366 | controller |     - '        '
2026-03-14 09:44:27.327370 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.327375 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.327379 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.327383 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.327387 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.327391 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.327396 | controller |       -- \'
2026-03-14 09:44:27.327400 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.327404 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.327410 | controller |     - '    logger.go:42: 09:31:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.327414 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.327418 | controller |     - '    logger.go:42: 09:31:31 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.327423 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.327427 | controller |       found'
2026-03-14 09:44:27.327431 | controller |     - '    logger.go:42: 09:31:32 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.327435 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.327440 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.327444 | controller |     - '        '
2026-03-14 09:44:27.327448 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.327452 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.327456 | controller |     - '        '
2026-03-14 09:44:27.327463 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.327467 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.327471 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.327476 | controller |     - '          exit 1'
2026-03-14 09:44:27.327480 | controller |     - '        fi'
2026-03-14 09:44:27.327484 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.327488 | controller |     - '        '
2026-03-14 09:44:27.327492 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.327497 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.327501 | controller |       | base64 -d)'
2026-03-14 09:44:27.327505 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.327509 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.327513 | controller |     - '          exit 1'
2026-03-14 09:44:27.327518 | controller |     - '        fi'
2026-03-14 09:44:27.327522 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.327526 | controller |     - '        '
2026-03-14 09:44:27.327530 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.327534 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.327539 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.327543 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.327547 | controller |       --timeout=300s'
2026-03-14 09:44:27.327552 | controller |     - '        '
2026-03-14 09:44:27.327558 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.327564 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.327572 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.327578 | controller |     - '        '
2026-03-14 09:44:27.327583 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.327588 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.327592 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.327596 | controller |     - '        '
2026-03-14 09:44:27.327600 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.327605 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.327617 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.327629 | controller |       available"'
2026-03-14 09:44:27.327635 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.327641 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.327647 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.327653 | controller |     - '          exit 1'
2026-03-14 09:44:27.327659 | controller |     - '        fi'
2026-03-14 09:44:27.327665 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.327670 | controller |     - '        '
2026-03-14 09:44:27.327675 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.327681 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.327686 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.327692 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.327697 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.327702 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.327706 | controller |       -- \'
2026-03-14 09:44:27.327710 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.327714 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.327719 | controller |     - '    logger.go:42: 09:31:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.327723 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.327727 | controller |     - '    logger.go:42: 09:31:32 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.327731 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.327736 | controller |       found'
2026-03-14 09:44:27.327774 | controller |     - '    logger.go:42: 09:31:33 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.327781 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.327785 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.327790 | controller |     - '        '
2026-03-14 09:44:27.327794 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.327798 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.327802 | controller |     - '        '
2026-03-14 09:44:27.327807 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.327811 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.327818 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.327822 | controller |     - '          exit 1'
2026-03-14 09:44:27.327826 | controller |     - '        fi'
2026-03-14 09:44:27.327831 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.327835 | controller |     - '        '
2026-03-14 09:44:27.327845 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.327849 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.327854 | controller |       | base64 -d)'
2026-03-14 09:44:27.327859 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.327863 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.327868 | controller |     - '          exit 1'
2026-03-14 09:44:27.327872 | controller |     - '        fi'
2026-03-14 09:44:27.327876 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.327882 | controller |     - '        '
2026-03-14 09:44:27.327887 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.327891 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.327895 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.327900 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.327904 | controller |       --timeout=300s'
2026-03-14 09:44:27.327908 | controller |     - '        '
2026-03-14 09:44:27.327912 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.327916 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.327921 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.327925 | controller |     - '        '
2026-03-14 09:44:27.327929 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.327933 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.327938 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.327942 | controller |     - '        '
2026-03-14 09:44:27.327946 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.327950 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.327961 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.327966 | controller |       available"'
2026-03-14 09:44:27.327970 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.327975 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.327979 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.327983 | controller |     - '          exit 1'
2026-03-14 09:44:27.327987 | controller |     - '        fi'
2026-03-14 09:44:27.327992 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.327996 | controller |     - '        '
2026-03-14 09:44:27.328000 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.328004 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.328009 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.328013 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.328017 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.328021 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.328026 | controller |       -- \'
2026-03-14 09:44:27.328030 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.328034 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.328039 | controller |     - '    logger.go:42: 09:31:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.328044 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.328048 | controller |     - '    logger.go:42: 09:31:33 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.328052 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.328057 | controller |       found'
2026-03-14 09:44:27.328061 | controller |     - '    logger.go:42: 09:31:34 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.328068 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.328073 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.328077 | controller |     - '        '
2026-03-14 09:44:27.328081 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.328085 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.328090 | controller |     - '        '
2026-03-14 09:44:27.328094 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.328098 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.328102 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.328107 | controller |     - '          exit 1'
2026-03-14 09:44:27.328111 | controller |     - '        fi'
2026-03-14 09:44:27.328115 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.328121 | controller |     - '        '
2026-03-14 09:44:27.328125 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.328130 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.328134 | controller |       | base64 -d)'
2026-03-14 09:44:27.328139 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.328144 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.328148 | controller |     - '          exit 1'
2026-03-14 09:44:27.328153 | controller |     - '        fi'
2026-03-14 09:44:27.328157 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.328161 | controller |     - '        '
2026-03-14 09:44:27.328165 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.328170 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.328174 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.328178 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.328182 | controller |       --timeout=300s'
2026-03-14 09:44:27.328186 | controller |     - '        '
2026-03-14 09:44:27.328191 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.328195 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.328199 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.328203 | controller |     - '        '
2026-03-14 09:44:27.328207 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.328212 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.328216 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.328220 | controller |     - '        '
2026-03-14 09:44:27.328224 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.328228 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.328233 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.328242 | controller |       available"'
2026-03-14 09:44:27.328247 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.328251 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.328255 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.328260 | controller |     - '          exit 1'
2026-03-14 09:44:27.328264 | controller |     - '        fi'
2026-03-14 09:44:27.328270 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.328274 | controller |     - '        '
2026-03-14 09:44:27.328278 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.328283 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.328287 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.328307 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.328311 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.328316 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.328320 | controller |       -- \'
2026-03-14 09:44:27.328324 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.328328 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.328333 | controller |     - '    logger.go:42: 09:31:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.328337 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.328341 | controller |     - '    logger.go:42: 09:31:35 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.328345 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.328350 | controller |       found'
2026-03-14 09:44:27.328354 | controller |     - '    logger.go:42: 09:31:36 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.328358 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.328362 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.328367 | controller |     - '        '
2026-03-14 09:44:27.328371 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.328375 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.328379 | controller |     - '        '
2026-03-14 09:44:27.328384 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.328388 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.328392 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.328396 | controller |     - '          exit 1'
2026-03-14 09:44:27.328401 | controller |     - '        fi'
2026-03-14 09:44:27.328405 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.328409 | controller |     - '        '
2026-03-14 09:44:27.328413 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.328418 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.328422 | controller |       | base64 -d)'
2026-03-14 09:44:27.328426 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.328432 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.328437 | controller |     - '          exit 1'
2026-03-14 09:44:27.328441 | controller |     - '        fi'
2026-03-14 09:44:27.328445 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.328449 | controller |     - '        '
2026-03-14 09:44:27.328453 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.328458 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.328462 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.328466 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.328470 | controller |       --timeout=300s'
2026-03-14 09:44:27.328474 | controller |     - '        '
2026-03-14 09:44:27.328480 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.328485 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.328489 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.328493 | controller |     - '        '
2026-03-14 09:44:27.328497 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.328501 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.328506 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.328510 | controller |     - '        '
2026-03-14 09:44:27.328514 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.328518 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.328522 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.328527 | controller |       available"'
2026-03-14 09:44:27.328537 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.328542 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.328546 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.328550 | controller |     - '          exit 1'
2026-03-14 09:44:27.328554 | controller |     - '        fi'
2026-03-14 09:44:27.328559 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.328563 | controller |     - '        '
2026-03-14 09:44:27.328567 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.328571 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.328575 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.328580 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.328586 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.328590 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.328595 | controller |       -- \'
2026-03-14 09:44:27.328599 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.328603 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.328607 | controller |     - '    logger.go:42: 09:31:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.328612 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.328617 | controller |     - '    logger.go:42: 09:31:36 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.328622 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.328628 | controller |       found'
2026-03-14 09:44:27.328634 | controller |     - '    logger.go:42: 09:31:37 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.328640 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.328646 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.328652 | controller |     - '        '
2026-03-14 09:44:27.328656 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.328660 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.328692 | controller |     - '        '
2026-03-14 09:44:27.328698 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.328705 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.328709 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.328713 | controller |     - '          exit 1'
2026-03-14 09:44:27.328717 | controller |     - '        fi'
2026-03-14 09:44:27.328722 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.328726 | controller |     - '        '
2026-03-14 09:44:27.328730 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.328734 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.328739 | controller |       | base64 -d)'
2026-03-14 09:44:27.328743 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.328747 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.328751 | controller |     - '          exit 1'
2026-03-14 09:44:27.328756 | controller |     - '        fi'
2026-03-14 09:44:27.328760 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.328764 | controller |     - '        '
2026-03-14 09:44:27.328769 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.328773 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.328779 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.328783 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.328787 | controller |       --timeout=300s'
2026-03-14 09:44:27.328793 | controller |     - '        '
2026-03-14 09:44:27.328797 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.328802 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.328806 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.328810 | controller |     - '        '
2026-03-14 09:44:27.328814 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.328818 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.328823 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.328827 | controller |     - '        '
2026-03-14 09:44:27.328831 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.328835 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.328839 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.328844 | controller |       available"'
2026-03-14 09:44:27.328848 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.328859 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.328864 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.328868 | controller |     - '          exit 1'
2026-03-14 09:44:27.328872 | controller |     - '        fi'
2026-03-14 09:44:27.328877 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.328881 | controller |     - '        '
2026-03-14 09:44:27.328885 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.328889 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.328894 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.328898 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.328904 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.328908 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.328912 | controller |       -- \'
2026-03-14 09:44:27.328919 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.328926 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.328932 | controller |     - '    logger.go:42: 09:31:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.328939 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.328948 | controller |     - '    logger.go:42: 09:31:37 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.328954 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.328960 | controller |       found'
2026-03-14 09:44:27.328966 | controller |     - '    logger.go:42: 09:31:38 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.328972 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.328978 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.328983 | controller |     - '        '
2026-03-14 09:44:27.328989 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.328994 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.329000 | controller |     - '        '
2026-03-14 09:44:27.329005 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.329010 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.329015 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.329021 | controller |     - '          exit 1'
2026-03-14 09:44:27.329026 | controller |     - '        fi'
2026-03-14 09:44:27.329030 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.329035 | controller |     - '        '
2026-03-14 09:44:27.329039 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.329043 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.329047 | controller |       | base64 -d)'
2026-03-14 09:44:27.329052 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.329056 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.329060 | controller |     - '          exit 1'
2026-03-14 09:44:27.329064 | controller |     - '        fi'
2026-03-14 09:44:27.329068 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.329073 | controller |     - '        '
2026-03-14 09:44:27.329077 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.329081 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.329085 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.329089 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.329094 | controller |       --timeout=300s'
2026-03-14 09:44:27.329098 | controller |     - '        '
2026-03-14 09:44:27.329102 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.329106 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.329110 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.329115 | controller |     - '        '
2026-03-14 09:44:27.329119 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.329125 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.329129 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.329134 | controller |     - '        '
2026-03-14 09:44:27.329138 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.329142 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.329146 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.329150 | controller |       available"'
2026-03-14 09:44:27.329155 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.329161 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.329173 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.329177 | controller |     - '          exit 1'
2026-03-14 09:44:27.329182 | controller |     - '        fi'
2026-03-14 09:44:27.329186 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.329216 | controller |     - '        '
2026-03-14 09:44:27.329220 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.329225 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.329229 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.329233 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.329237 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.329242 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.329246 | controller |       -- \'
2026-03-14 09:44:27.329250 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.329254 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.329258 | controller |     - '    logger.go:42: 09:31:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.329263 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.329267 | controller |     - '    logger.go:42: 09:31:38 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.329271 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.329275 | controller |       found'
2026-03-14 09:44:27.329280 | controller |     - '    logger.go:42: 09:31:39 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.329284 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.329302 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.329309 | controller |     - '        '
2026-03-14 09:44:27.329314 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.329318 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.329323 | controller |     - '        '
2026-03-14 09:44:27.329327 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.329331 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.329336 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.329340 | controller |     - '          exit 1'
2026-03-14 09:44:27.329344 | controller |     - '        fi'
2026-03-14 09:44:27.329349 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.329353 | controller |     - '        '
2026-03-14 09:44:27.329357 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.329361 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.329383 | controller |       | base64 -d)'
2026-03-14 09:44:27.329387 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.329392 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.329396 | controller |     - '          exit 1'
2026-03-14 09:44:27.329400 | controller |     - '        fi'
2026-03-14 09:44:27.329404 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.329409 | controller |     - '        '
2026-03-14 09:44:27.329413 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.329417 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.329421 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.329430 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.329434 | controller |       --timeout=300s'
2026-03-14 09:44:27.329438 | controller |     - '        '
2026-03-14 09:44:27.329443 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.329447 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.329451 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.329455 | controller |     - '        '
2026-03-14 09:44:27.329460 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.329464 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.329468 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.329472 | controller |     - '        '
2026-03-14 09:44:27.329476 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.329481 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.329486 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.329490 | controller |       available"'
2026-03-14 09:44:27.329494 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.329500 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.329512 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.329516 | controller |     - '          exit 1'
2026-03-14 09:44:27.329521 | controller |     - '        fi'
2026-03-14 09:44:27.329525 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.329529 | controller |     - '        '
2026-03-14 09:44:27.329533 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.329537 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.329542 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.329546 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.329550 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.329554 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.329558 | controller |       -- \'
2026-03-14 09:44:27.329563 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.329567 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.329573 | controller |     - '    logger.go:42: 09:31:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.329577 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.329581 | controller |     - '    logger.go:42: 09:31:39 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.329585 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.329590 | controller |       found'
2026-03-14 09:44:27.329594 | controller |     - '    logger.go:42: 09:31:40 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.329598 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.329602 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.329607 | controller |     - '        '
2026-03-14 09:44:27.329611 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.329615 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.329621 | controller |     - '        '
2026-03-14 09:44:27.329627 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.329633 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.329639 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.329645 | controller |     - '          exit 1'
2026-03-14 09:44:27.329651 | controller |     - '        fi'
2026-03-14 09:44:27.329655 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.329659 | controller |     - '        '
2026-03-14 09:44:27.329663 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.329668 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.329672 | controller |       | base64 -d)'
2026-03-14 09:44:27.329676 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.329680 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.329684 | controller |     - '          exit 1'
2026-03-14 09:44:27.329689 | controller |     - '        fi'
2026-03-14 09:44:27.329693 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.329697 | controller |     - '        '
2026-03-14 09:44:27.329701 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.329705 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.329709 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.329714 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.329718 | controller |       --timeout=300s'
2026-03-14 09:44:27.329722 | controller |     - '        '
2026-03-14 09:44:27.329726 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.329730 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.329735 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.329739 | controller |     - '        '
2026-03-14 09:44:27.329743 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.329749 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.329754 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.329758 | controller |     - '        '
2026-03-14 09:44:27.329762 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.329766 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.329770 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.329776 | controller |       available"'
2026-03-14 09:44:27.329781 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.329785 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.329789 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.329793 | controller |     - '          exit 1'
2026-03-14 09:44:27.329798 | controller |     - '        fi'
2026-03-14 09:44:27.329808 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.329813 | controller |     - '        '
2026-03-14 09:44:27.329817 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.329822 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.329826 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.329830 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.329834 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.329838 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.329842 | controller |       -- \'
2026-03-14 09:44:27.329847 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.329851 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.329857 | controller |     - '    logger.go:42: 09:31:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.329885 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.329891 | controller |     - '    logger.go:42: 09:31:40 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.329896 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.329900 | controller |       found'
2026-03-14 09:44:27.329904 | controller |     - '    logger.go:42: 09:31:41 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.329909 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.329913 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.329917 | controller |     - '        '
2026-03-14 09:44:27.329923 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.329928 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.329932 | controller |     - '        '
2026-03-14 09:44:27.329937 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.329942 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.329947 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.329952 | controller |     - '          exit 1'
2026-03-14 09:44:27.329956 | controller |     - '        fi'
2026-03-14 09:44:27.329960 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.329964 | controller |     - '        '
2026-03-14 09:44:27.329985 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.329989 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.329994 | controller |       | base64 -d)'
2026-03-14 09:44:27.329998 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.330002 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.330006 | controller |     - '          exit 1'
2026-03-14 09:44:27.330011 | controller |     - '        fi'
2026-03-14 09:44:27.330015 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.330019 | controller |     - '        '
2026-03-14 09:44:27.330023 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.330030 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.330034 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.330038 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.330043 | controller |       --timeout=300s'
2026-03-14 09:44:27.330047 | controller |     - '        '
2026-03-14 09:44:27.330051 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.330055 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.330060 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.330064 | controller |     - '        '
2026-03-14 09:44:27.330070 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.330074 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.330079 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.330083 | controller |     - '        '
2026-03-14 09:44:27.330088 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.330093 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.330097 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.330101 | controller |       available"'
2026-03-14 09:44:27.330105 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.330110 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.330114 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.330118 | controller |     - '          exit 1'
2026-03-14 09:44:27.330122 | controller |     - '        fi'
2026-03-14 09:44:27.330133 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.330138 | controller |     - '        '
2026-03-14 09:44:27.330142 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.330147 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.330151 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.330155 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.330159 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.330163 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.330168 | controller |       -- \'
2026-03-14 09:44:27.330172 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.330176 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.330180 | controller |     - '    logger.go:42: 09:31:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.330184 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.330189 | controller |     - '    logger.go:42: 09:31:41 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.330193 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.330197 | controller |       found'
2026-03-14 09:44:27.330201 | controller |     - '    logger.go:42: 09:31:42 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.330207 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.330211 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.330215 | controller |     - '        '
2026-03-14 09:44:27.330220 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.330224 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.330228 | controller |     - '        '
2026-03-14 09:44:27.330232 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.330236 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.330241 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.330245 | controller |     - '          exit 1'
2026-03-14 09:44:27.330249 | controller |     - '        fi'
2026-03-14 09:44:27.330253 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.330258 | controller |     - '        '
2026-03-14 09:44:27.330262 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.330266 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.330270 | controller |       | base64 -d)'
2026-03-14 09:44:27.330274 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.330279 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.330283 | controller |     - '          exit 1'
2026-03-14 09:44:27.330287 | controller |     - '        fi'
2026-03-14 09:44:27.330305 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.330310 | controller |     - '        '
2026-03-14 09:44:27.330314 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.330318 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.330322 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.330326 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.330331 | controller |       --timeout=300s'
2026-03-14 09:44:27.330335 | controller |     - '        '
2026-03-14 09:44:27.330339 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.330343 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.330347 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.330352 | controller |     - '        '
2026-03-14 09:44:27.330356 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.330360 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.330364 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.330368 | controller |     - '        '
2026-03-14 09:44:27.330373 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.330377 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.330381 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.330385 | controller |       available"'
2026-03-14 09:44:27.330389 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.330393 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.330397 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.330402 | controller |     - '          exit 1'
2026-03-14 09:44:27.330406 | controller |     - '        fi'
2026-03-14 09:44:27.330410 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.330416 | controller |     - '        '
2026-03-14 09:44:27.330439 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.330443 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.330447 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.330452 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.330457 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.330461 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.330465 | controller |       -- \'
2026-03-14 09:44:27.330469 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.330474 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.330478 | controller |     - '    logger.go:42: 09:31:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.330482 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.330486 | controller |     - '    logger.go:42: 09:31:43 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.330490 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.330495 | controller |       found'
2026-03-14 09:44:27.330499 | controller |     - '    logger.go:42: 09:31:44 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.330503 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.330507 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.330511 | controller |     - '        '
2026-03-14 09:44:27.330515 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.330520 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.330524 | controller |     - '        '
2026-03-14 09:44:27.330528 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.330532 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.330536 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.330541 | controller |     - '          exit 1'
2026-03-14 09:44:27.330545 | controller |     - '        fi'
2026-03-14 09:44:27.330549 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.330553 | controller |     - '        '
2026-03-14 09:44:27.330557 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.330562 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.330566 | controller |       | base64 -d)'
2026-03-14 09:44:27.330570 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.330574 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.330578 | controller |     - '          exit 1'
2026-03-14 09:44:27.330583 | controller |     - '        fi'
2026-03-14 09:44:27.330587 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.330591 | controller |     - '        '
2026-03-14 09:44:27.330595 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.330599 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.330603 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.330608 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.330612 | controller |       --timeout=300s'
2026-03-14 09:44:27.330617 | controller |     - '        '
2026-03-14 09:44:27.330623 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.330632 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.330638 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.330644 | controller |     - '        '
2026-03-14 09:44:27.330649 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.330653 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.330657 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.330662 | controller |     - '        '
2026-03-14 09:44:27.330666 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.330670 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.330677 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.330681 | controller |       available"'
2026-03-14 09:44:27.330685 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.330689 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.330694 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.330698 | controller |     - '          exit 1'
2026-03-14 09:44:27.330702 | controller |     - '        fi'
2026-03-14 09:44:27.330706 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.330710 | controller |     - '        '
2026-03-14 09:44:27.330721 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.330726 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.330730 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.330734 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.330738 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.330742 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.330749 | controller |       -- \'
2026-03-14 09:44:27.330753 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.330757 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.330761 | controller |     - '    logger.go:42: 09:31:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.330765 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.330770 | controller |     - '    logger.go:42: 09:31:44 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.330774 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.330778 | controller |       found'
2026-03-14 09:44:27.330782 | controller |     - '    logger.go:42: 09:31:45 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.330787 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.330791 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.330795 | controller |     - '        '
2026-03-14 09:44:27.330799 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.330803 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.330807 | controller |     - '        '
2026-03-14 09:44:27.330812 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.330816 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.330822 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.330826 | controller |     - '          exit 1'
2026-03-14 09:44:27.330830 | controller |     - '        fi'
2026-03-14 09:44:27.330835 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.330839 | controller |     - '        '
2026-03-14 09:44:27.330847 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.330851 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.330855 | controller |       | base64 -d)'
2026-03-14 09:44:27.330859 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.330864 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.330868 | controller |     - '          exit 1'
2026-03-14 09:44:27.330872 | controller |     - '        fi'
2026-03-14 09:44:27.330876 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.330881 | controller |     - '        '
2026-03-14 09:44:27.330885 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.330889 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.330893 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.330897 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.330902 | controller |       --timeout=300s'
2026-03-14 09:44:27.330906 | controller |     - '        '
2026-03-14 09:44:27.330910 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.330914 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.330918 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.330923 | controller |     - '        '
2026-03-14 09:44:27.330927 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.330931 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.330935 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.330940 | controller |     - '        '
2026-03-14 09:44:27.330944 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.330948 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.330952 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.330956 | controller |       available"'
2026-03-14 09:44:27.330960 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.330965 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.330969 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.330973 | controller |     - '          exit 1'
2026-03-14 09:44:27.330977 | controller |     - '        fi'
2026-03-14 09:44:27.330982 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.330986 | controller |     - '        '
2026-03-14 09:44:27.330990 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.331000 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.331004 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.331008 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.331013 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.331018 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.331023 | controller |       -- \'
2026-03-14 09:44:27.331027 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.331031 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.331035 | controller |     - '    logger.go:42: 09:31:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.331039 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.331044 | controller |     - '    logger.go:42: 09:31:45 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.331048 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.331052 | controller |       found'
2026-03-14 09:44:27.331056 | controller |     - '    logger.go:42: 09:31:46 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.331060 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.331065 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.331070 | controller |     - '        '
2026-03-14 09:44:27.331075 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.331081 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.331087 | controller |     - '        '
2026-03-14 09:44:27.331093 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.331098 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.331104 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.331109 | controller |     - '          exit 1'
2026-03-14 09:44:27.331115 | controller |     - '        fi'
2026-03-14 09:44:27.331121 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.331127 | controller |     - '        '
2026-03-14 09:44:27.331133 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.331138 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.331144 | controller |       | base64 -d)'
2026-03-14 09:44:27.331150 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.331156 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.331161 | controller |     - '          exit 1'
2026-03-14 09:44:27.331170 | controller |     - '        fi'
2026-03-14 09:44:27.331176 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.331182 | controller |     - '        '
2026-03-14 09:44:27.331187 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.331193 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.331199 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.331205 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.331211 | controller |       --timeout=300s'
2026-03-14 09:44:27.331218 | controller |     - '        '
2026-03-14 09:44:27.331224 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.331230 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.331235 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.331239 | controller |     - '        '
2026-03-14 09:44:27.331245 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.331250 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.331256 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.331260 | controller |     - '        '
2026-03-14 09:44:27.331264 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.331269 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.331273 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.331277 | controller |       available"'
2026-03-14 09:44:27.331281 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.331306 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.331314 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.331318 | controller |     - '          exit 1'
2026-03-14 09:44:27.331323 | controller |     - '        fi'
2026-03-14 09:44:27.331327 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.331331 | controller |     - '        '
2026-03-14 09:44:27.331335 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.331349 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.331355 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.331361 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.331367 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.331373 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.331379 | controller |       -- \'
2026-03-14 09:44:27.331384 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.331389 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.331393 | controller |     - '    logger.go:42: 09:31:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.331397 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.331401 | controller |     - '    logger.go:42: 09:31:46 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.331406 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.331411 | controller |       found'
2026-03-14 09:44:27.331417 | controller |     - '    logger.go:42: 09:31:47 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.331423 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.331429 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.331434 | controller |     - '        '
2026-03-14 09:44:27.331440 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.331446 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.331452 | controller |     - '        '
2026-03-14 09:44:27.331457 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.331463 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.331469 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.331475 | controller |     - '          exit 1'
2026-03-14 09:44:27.331481 | controller |     - '        fi'
2026-03-14 09:44:27.331486 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.331492 | controller |     - '        '
2026-03-14 09:44:27.331498 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.331503 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.331513 | controller |       | base64 -d)'
2026-03-14 09:44:27.331519 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.331525 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.331531 | controller |     - '          exit 1'
2026-03-14 09:44:27.331536 | controller |     - '        fi'
2026-03-14 09:44:27.331542 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.331548 | controller |     - '        '
2026-03-14 09:44:27.331552 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.331556 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.331560 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.331565 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.331569 | controller |       --timeout=300s'
2026-03-14 09:44:27.331573 | controller |     - '        '
2026-03-14 09:44:27.331577 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.331581 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.331586 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.331590 | controller |     - '        '
2026-03-14 09:44:27.331594 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.331598 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.331602 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.331607 | controller |     - '        '
2026-03-14 09:44:27.331611 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.331615 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.331621 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.331627 | controller |       available"'
2026-03-14 09:44:27.331633 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.331639 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.331645 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.331651 | controller |     - '          exit 1'
2026-03-14 09:44:27.331657 | controller |     - '        fi'
2026-03-14 09:44:27.331662 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.331668 | controller |     - '        '
2026-03-14 09:44:27.331674 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.331680 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.331693 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.331700 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.331706 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.331711 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.331717 | controller |       -- \'
2026-03-14 09:44:27.331723 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.331729 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.331735 | controller |     - '    logger.go:42: 09:31:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.331746 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.331752 | controller |     - '    logger.go:42: 09:31:47 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.331758 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.331764 | controller |       found'
2026-03-14 09:44:27.331770 | controller |     - '    logger.go:42: 09:31:48 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.331776 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.331784 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.331790 | controller |     - '        '
2026-03-14 09:44:27.331796 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.331802 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.331808 | controller |     - '        '
2026-03-14 09:44:27.331814 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.331820 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.331826 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.331832 | controller |     - '          exit 1'
2026-03-14 09:44:27.331837 | controller |     - '        fi'
2026-03-14 09:44:27.331843 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.331849 | controller |     - '        '
2026-03-14 09:44:27.331855 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.331861 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.331866 | controller |       | base64 -d)'
2026-03-14 09:44:27.331872 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.331878 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.331883 | controller |     - '          exit 1'
2026-03-14 09:44:27.331889 | controller |     - '        fi'
2026-03-14 09:44:27.331895 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.331901 | controller |     - '        '
2026-03-14 09:44:27.331907 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.331912 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.331918 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.331924 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.331930 | controller |       --timeout=300s'
2026-03-14 09:44:27.331936 | controller |     - '        '
2026-03-14 09:44:27.331941 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.331947 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.331953 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.331958 | controller |     - '        '
2026-03-14 09:44:27.331964 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.331970 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.331979 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.331985 | controller |     - '        '
2026-03-14 09:44:27.331991 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.331996 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.332002 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.332011 | controller |       available"'
2026-03-14 09:44:27.332017 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.332022 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.332028 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.332034 | controller |     - '          exit 1'
2026-03-14 09:44:27.332040 | controller |     - '        fi'
2026-03-14 09:44:27.332046 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.332052 | controller |     - '        '
2026-03-14 09:44:27.332060 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.332066 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.332081 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.332088 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.332093 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.332099 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.332105 | controller |       -- \'
2026-03-14 09:44:27.332113 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.332119 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.332125 | controller |     - '    logger.go:42: 09:31:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.332131 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.332136 | controller |     - '    logger.go:42: 09:31:48 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.332142 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.332147 | controller |       found'
2026-03-14 09:44:27.332153 | controller |     - '    logger.go:42: 09:31:49 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.332159 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.332163 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.332167 | controller |     - '        '
2026-03-14 09:44:27.332172 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.332176 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.332180 | controller |     - '        '
2026-03-14 09:44:27.332184 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.332188 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.332193 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.332197 | controller |     - '          exit 1'
2026-03-14 09:44:27.332201 | controller |     - '        fi'
2026-03-14 09:44:27.332205 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.332209 | controller |     - '        '
2026-03-14 09:44:27.332214 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.332218 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.332222 | controller |       | base64 -d)'
2026-03-14 09:44:27.332226 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.332230 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.332235 | controller |     - '          exit 1'
2026-03-14 09:44:27.332239 | controller |     - '        fi'
2026-03-14 09:44:27.332243 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.332247 | controller |     - '        '
2026-03-14 09:44:27.332251 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.332258 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.332262 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.332266 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.332270 | controller |       --timeout=300s'
2026-03-14 09:44:27.332274 | controller |     - '        '
2026-03-14 09:44:27.332279 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.332283 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.332290 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.332311 | controller |     - '        '
2026-03-14 09:44:27.332316 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.332320 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.332324 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.332329 | controller |     - '        '
2026-03-14 09:44:27.332333 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.332337 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.332341 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.332346 | controller |       available"'
2026-03-14 09:44:27.332350 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.332356 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.332360 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.332365 | controller |     - '          exit 1'
2026-03-14 09:44:27.332369 | controller |     - '        fi'
2026-03-14 09:44:27.332373 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.332377 | controller |     - '        '
2026-03-14 09:44:27.332381 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.332386 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.332390 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.332401 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.332405 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.332410 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.332414 | controller |       -- \'
2026-03-14 09:44:27.332418 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.332422 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.332427 | controller |     - '    logger.go:42: 09:31:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.332431 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.332435 | controller |     - '    logger.go:42: 09:31:49 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.332439 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.332443 | controller |       found'
2026-03-14 09:44:27.332448 | controller |     - '    logger.go:42: 09:31:50 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.332452 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.332460 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.332464 | controller |     - '        '
2026-03-14 09:44:27.332468 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.332472 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.332477 | controller |     - '        '
2026-03-14 09:44:27.332483 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.332488 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.332492 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.332496 | controller |     - '          exit 1'
2026-03-14 09:44:27.332500 | controller |     - '        fi'
2026-03-14 09:44:27.332505 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.332509 | controller |     - '        '
2026-03-14 09:44:27.332513 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.332517 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.332521 | controller |       | base64 -d)'
2026-03-14 09:44:27.332526 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.332530 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.332534 | controller |     - '          exit 1'
2026-03-14 09:44:27.332538 | controller |     - '        fi'
2026-03-14 09:44:27.332543 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.332547 | controller |     - '        '
2026-03-14 09:44:27.332551 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.332555 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.332560 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.332564 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.332568 | controller |       --timeout=300s'
2026-03-14 09:44:27.332572 | controller |     - '        '
2026-03-14 09:44:27.332576 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.332580 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.332585 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.332589 | controller |     - '        '
2026-03-14 09:44:27.332593 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.332597 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.332601 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.332606 | controller |     - '        '
2026-03-14 09:44:27.332610 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.332614 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.332620 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.332626 | controller |       available"'
2026-03-14 09:44:27.332632 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.332638 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.332644 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.332649 | controller |     - '          exit 1'
2026-03-14 09:44:27.332655 | controller |     - '        fi'
2026-03-14 09:44:27.332661 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.332667 | controller |     - '        '
2026-03-14 09:44:27.332675 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.332680 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.332686 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.332694 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.332708 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.332713 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.332718 | controller |       -- \'
2026-03-14 09:44:27.332722 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.332726 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.332730 | controller |     - '    logger.go:42: 09:31:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.332735 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.332739 | controller |     - '    logger.go:42: 09:31:50 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.332743 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.332747 | controller |       found'
2026-03-14 09:44:27.332752 | controller |     - '    logger.go:42: 09:31:51 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.332756 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.332760 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.332764 | controller |     - '        '
2026-03-14 09:44:27.332768 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.332773 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.332777 | controller |     - '        '
2026-03-14 09:44:27.332781 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.332785 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.332789 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.332794 | controller |     - '          exit 1'
2026-03-14 09:44:27.332798 | controller |     - '        fi'
2026-03-14 09:44:27.332802 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.332806 | controller |     - '        '
2026-03-14 09:44:27.332810 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.332815 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.332819 | controller |       | base64 -d)'
2026-03-14 09:44:27.332823 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.332836 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.332840 | controller |     - '          exit 1'
2026-03-14 09:44:27.332844 | controller |     - '        fi'
2026-03-14 09:44:27.332849 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.332853 | controller |     - '        '
2026-03-14 09:44:27.332857 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.332861 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.332866 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.332870 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.332874 | controller |       --timeout=300s'
2026-03-14 09:44:27.332878 | controller |     - '        '
2026-03-14 09:44:27.332882 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.332887 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.332893 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.332897 | controller |     - '        '
2026-03-14 09:44:27.332901 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.332905 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.332910 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.332914 | controller |     - '        '
2026-03-14 09:44:27.332918 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.332922 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.332926 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.332931 | controller |       available"'
2026-03-14 09:44:27.332935 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.332939 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.332943 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.332947 | controller |     - '          exit 1'
2026-03-14 09:44:27.332952 | controller |     - '        fi'
2026-03-14 09:44:27.332956 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.332960 | controller |     - '        '
2026-03-14 09:44:27.332964 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.332968 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.332973 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.332977 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.332987 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.332991 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.332996 | controller |       -- \'
2026-03-14 09:44:27.333000 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.333004 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.333010 | controller |     - '    logger.go:42: 09:31:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.333014 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.333019 | controller |     - '    logger.go:42: 09:31:52 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.333023 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.333027 | controller |       found'
2026-03-14 09:44:27.333031 | controller |     - '    logger.go:42: 09:31:53 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.333035 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.333040 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.333044 | controller |     - '        '
2026-03-14 09:44:27.333048 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.333052 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.333056 | controller |     - '        '
2026-03-14 09:44:27.333061 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.333065 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.333069 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.333075 | controller |     - '          exit 1'
2026-03-14 09:44:27.333079 | controller |     - '        fi'
2026-03-14 09:44:27.333083 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.333088 | controller |     - '        '
2026-03-14 09:44:27.333092 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.333096 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.333100 | controller |       | base64 -d)'
2026-03-14 09:44:27.333104 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.333109 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.333113 | controller |     - '          exit 1'
2026-03-14 09:44:27.333117 | controller |     - '        fi'
2026-03-14 09:44:27.333121 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.333126 | controller |     - '        '
2026-03-14 09:44:27.333131 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.333135 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.333139 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.333143 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.333147 | controller |       --timeout=300s'
2026-03-14 09:44:27.333152 | controller |     - '        '
2026-03-14 09:44:27.333156 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.333160 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.333164 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.333169 | controller |     - '        '
2026-03-14 09:44:27.333173 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.333177 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.333181 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.333185 | controller |     - '        '
2026-03-14 09:44:27.333189 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.333194 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.333198 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.333202 | controller |       available"'
2026-03-14 09:44:27.333206 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.333210 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.333215 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.333219 | controller |     - '          exit 1'
2026-03-14 09:44:27.333224 | controller |     - '        fi'
2026-03-14 09:44:27.333229 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.333233 | controller |     - '        '
2026-03-14 09:44:27.333237 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.333241 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.333245 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.333250 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.333259 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.333265 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.333270 | controller |       -- \'
2026-03-14 09:44:27.333274 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.333278 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.333282 | controller |     - '    logger.go:42: 09:31:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.333289 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.333309 | controller |     - '    logger.go:42: 09:31:53 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.333314 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.333318 | controller |       found'
2026-03-14 09:44:27.333322 | controller |     - '    logger.go:42: 09:31:54 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.333326 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.333331 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.333335 | controller |     - '        '
2026-03-14 09:44:27.333339 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.333343 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.333347 | controller |     - '        '
2026-03-14 09:44:27.333352 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.333356 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.333360 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.333364 | controller |     - '          exit 1'
2026-03-14 09:44:27.333368 | controller |     - '        fi'
2026-03-14 09:44:27.333373 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.333377 | controller |     - '        '
2026-03-14 09:44:27.333381 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.333385 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.333389 | controller |       | base64 -d)'
2026-03-14 09:44:27.333394 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.333398 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.333402 | controller |     - '          exit 1'
2026-03-14 09:44:27.333406 | controller |     - '        fi'
2026-03-14 09:44:27.333410 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.333415 | controller |     - '        '
2026-03-14 09:44:27.333419 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.333423 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.333427 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.333431 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.333436 | controller |       --timeout=300s'
2026-03-14 09:44:27.333440 | controller |     - '        '
2026-03-14 09:44:27.333444 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.333448 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.333452 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.333457 | controller |     - '        '
2026-03-14 09:44:27.333461 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.333465 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.333471 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.333475 | controller |     - '        '
2026-03-14 09:44:27.333479 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.333483 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.333488 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.333492 | controller |       available"'
2026-03-14 09:44:27.333498 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.333502 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.333506 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.333510 | controller |     - '          exit 1'
2026-03-14 09:44:27.333515 | controller |     - '        fi'
2026-03-14 09:44:27.333519 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.333523 | controller |     - '        '
2026-03-14 09:44:27.333527 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.333531 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.333536 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.333540 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.333544 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.333554 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.333559 | controller |       -- \'
2026-03-14 09:44:27.333563 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.333567 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.333571 | controller |     - '    logger.go:42: 09:31:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.333575 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.333580 | controller |     - '    logger.go:42: 09:31:54 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.333584 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.333588 | controller |       found'
2026-03-14 09:44:27.333592 | controller |     - '    logger.go:42: 09:31:55 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.333597 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.333601 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.333605 | controller |     - '        '
2026-03-14 09:44:27.333609 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.333613 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.333618 | controller |     - '        '
2026-03-14 09:44:27.333622 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.333627 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.333632 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.333638 | controller |     - '          exit 1'
2026-03-14 09:44:27.333644 | controller |     - '        fi'
2026-03-14 09:44:27.333650 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.333655 | controller |     - '        '
2026-03-14 09:44:27.333662 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.333666 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.333671 | controller |       | base64 -d)'
2026-03-14 09:44:27.333675 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.333681 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.333685 | controller |     - '          exit 1'
2026-03-14 09:44:27.333689 | controller |     - '        fi'
2026-03-14 09:44:27.333693 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.333698 | controller |     - '        '
2026-03-14 09:44:27.333702 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.333706 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.333710 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.333715 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.333719 | controller |       --timeout=300s'
2026-03-14 09:44:27.333723 | controller |     - '        '
2026-03-14 09:44:27.333728 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.333732 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.333736 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.333741 | controller |     - '        '
2026-03-14 09:44:27.333745 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.333749 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.333753 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.333758 | controller |     - '        '
2026-03-14 09:44:27.333762 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.333766 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.333770 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.333774 | controller |       available"'
2026-03-14 09:44:27.333779 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.333783 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.333787 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.333791 | controller |     - '          exit 1'
2026-03-14 09:44:27.333795 | controller |     - '        fi'
2026-03-14 09:44:27.333800 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.333804 | controller |     - '        '
2026-03-14 09:44:27.333808 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.333812 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.333816 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.333820 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.333825 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.333838 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.333843 | controller |       -- \'
2026-03-14 09:44:27.333847 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.333852 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.333856 | controller |     - '    logger.go:42: 09:31:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.333866 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.333870 | controller |     - '    logger.go:42: 09:31:55 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.333874 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.333878 | controller |       found'
2026-03-14 09:44:27.333883 | controller |     - '    logger.go:42: 09:31:56 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.333887 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.333891 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.333895 | controller |     - '        '
2026-03-14 09:44:27.333899 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.333904 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.333908 | controller |     - '        '
2026-03-14 09:44:27.333912 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.333916 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.333920 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.333924 | controller |     - '          exit 1'
2026-03-14 09:44:27.333930 | controller |     - '        fi'
2026-03-14 09:44:27.333936 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.333942 | controller |     - '        '
2026-03-14 09:44:27.333947 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.333954 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.334007 | controller |       | base64 -d)'
2026-03-14 09:44:27.334014 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.334020 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.334026 | controller |     - '          exit 1'
2026-03-14 09:44:27.334031 | controller |     - '        fi'
2026-03-14 09:44:27.334037 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.334041 | controller |     - '        '
2026-03-14 09:44:27.334046 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.334050 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.334054 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.334058 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.334062 | controller |       --timeout=300s'
2026-03-14 09:44:27.334067 | controller |     - '        '
2026-03-14 09:44:27.334071 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.334075 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.334079 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.334084 | controller |     - '        '
2026-03-14 09:44:27.334088 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.334092 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.334096 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.334100 | controller |     - '        '
2026-03-14 09:44:27.334104 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.334112 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.334116 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.334122 | controller |       available"'
2026-03-14 09:44:27.334127 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.334136 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.334142 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.334148 | controller |     - '          exit 1'
2026-03-14 09:44:27.334153 | controller |     - '        fi'
2026-03-14 09:44:27.334159 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.334165 | controller |     - '        '
2026-03-14 09:44:27.334171 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.334175 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.334180 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.334184 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.334188 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.334192 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.334197 | controller |       -- \'
2026-03-14 09:44:27.334211 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.334218 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.334227 | controller |     - '    logger.go:42: 09:31:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.334233 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.334239 | controller |     - '    logger.go:42: 09:31:56 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.334246 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.334252 | controller |       found'
2026-03-14 09:44:27.334257 | controller |     - '    logger.go:42: 09:31:57 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.334263 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.334269 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.334275 | controller |     - '        '
2026-03-14 09:44:27.334281 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.334287 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.334312 | controller |     - '        '
2026-03-14 09:44:27.334319 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.334325 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.334330 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.334336 | controller |     - '          exit 1'
2026-03-14 09:44:27.334342 | controller |     - '        fi'
2026-03-14 09:44:27.334352 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.334358 | controller |     - '        '
2026-03-14 09:44:27.334364 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.334370 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.334375 | controller |       | base64 -d)'
2026-03-14 09:44:27.334381 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.334387 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.334391 | controller |     - '          exit 1'
2026-03-14 09:44:27.334396 | controller |     - '        fi'
2026-03-14 09:44:27.334400 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.334404 | controller |     - '        '
2026-03-14 09:44:27.334408 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.334413 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.334422 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.334427 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.334433 | controller |       --timeout=300s'
2026-03-14 09:44:27.334439 | controller |     - '        '
2026-03-14 09:44:27.334445 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.334451 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.334457 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.334463 | controller |     - '        '
2026-03-14 09:44:27.334469 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.334475 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.334480 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.334486 | controller |     - '        '
2026-03-14 09:44:27.334492 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.334498 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.334503 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.334509 | controller |       available"'
2026-03-14 09:44:27.334515 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.334519 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.334524 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.334528 | controller |     - '          exit 1'
2026-03-14 09:44:27.334532 | controller |     - '        fi'
2026-03-14 09:44:27.334536 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.334540 | controller |     - '        '
2026-03-14 09:44:27.334544 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.334549 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.334553 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.334557 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.334561 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.334565 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.334570 | controller |       -- \'
2026-03-14 09:44:27.334583 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.334587 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.334592 | controller |     - '    logger.go:42: 09:31:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.334596 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.334600 | controller |     - '    logger.go:42: 09:31:57 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.334607 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.334611 | controller |       found'
2026-03-14 09:44:27.334615 | controller |     - '    logger.go:42: 09:31:58 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.334620 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.334624 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.334628 | controller |     - '        '
2026-03-14 09:44:27.334634 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.334638 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.334643 | controller |     - '        '
2026-03-14 09:44:27.334647 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.334651 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.334655 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.334660 | controller |     - '          exit 1'
2026-03-14 09:44:27.334664 | controller |     - '        fi'
2026-03-14 09:44:27.334668 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.334672 | controller |     - '        '
2026-03-14 09:44:27.334676 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.334681 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.334685 | controller |       | base64 -d)'
2026-03-14 09:44:27.334689 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.334693 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.334697 | controller |     - '          exit 1'
2026-03-14 09:44:27.334702 | controller |     - '        fi'
2026-03-14 09:44:27.334706 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.334710 | controller |     - '        '
2026-03-14 09:44:27.334716 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.334720 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.334725 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.334729 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.334733 | controller |       --timeout=300s'
2026-03-14 09:44:27.334737 | controller |     - '        '
2026-03-14 09:44:27.334742 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.334746 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.334750 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.334754 | controller |     - '        '
2026-03-14 09:44:27.334758 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.334763 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.334768 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.334772 | controller |     - '        '
2026-03-14 09:44:27.334776 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.334780 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.334785 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.334789 | controller |       available"'
2026-03-14 09:44:27.334793 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.334797 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.334801 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.334806 | controller |     - '          exit 1'
2026-03-14 09:44:27.334810 | controller |     - '        fi'
2026-03-14 09:44:27.334814 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.334818 | controller |     - '        '
2026-03-14 09:44:27.334822 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.334828 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.334832 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.334837 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.334841 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.334845 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.334849 | controller |       -- \'
2026-03-14 09:44:27.334853 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.334864 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.334868 | controller |     - '    logger.go:42: 09:31:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.334873 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.334877 | controller |     - '    logger.go:42: 09:31:59 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.334881 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.334885 | controller |       found'
2026-03-14 09:44:27.334890 | controller |     - '    logger.go:42: 09:32:00 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.334894 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.334898 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.334902 | controller |     - '        '
2026-03-14 09:44:27.334907 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.334911 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.334915 | controller |     - '        '
2026-03-14 09:44:27.334919 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.334923 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.334927 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.334932 | controller |     - '          exit 1'
2026-03-14 09:44:27.334936 | controller |     - '        fi'
2026-03-14 09:44:27.334940 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.334944 | controller |     - '        '
2026-03-14 09:44:27.334949 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.334953 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.334957 | controller |       | base64 -d)'
2026-03-14 09:44:27.334961 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.334966 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.334970 | controller |     - '          exit 1'
2026-03-14 09:44:27.334974 | controller |     - '        fi'
2026-03-14 09:44:27.334978 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.334982 | controller |     - '        '
2026-03-14 09:44:27.334986 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.334991 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.334995 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.334999 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.335003 | controller |       --timeout=300s'
2026-03-14 09:44:27.335007 | controller |     - '        '
2026-03-14 09:44:27.335012 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.335016 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.335022 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.335026 | controller |     - '        '
2026-03-14 09:44:27.335030 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.335034 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.335038 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.335043 | controller |     - '        '
2026-03-14 09:44:27.335047 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.335051 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.335055 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.335059 | controller |       available"'
2026-03-14 09:44:27.335064 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.335070 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.335074 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.335079 | controller |     - '          exit 1'
2026-03-14 09:44:27.335083 | controller |     - '        fi'
2026-03-14 09:44:27.335087 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.335091 | controller |     - '        '
2026-03-14 09:44:27.335095 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.335100 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.335104 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.335108 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.335112 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.335116 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.335120 | controller |       -- \'
2026-03-14 09:44:27.335125 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.335134 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.335139 | controller |     - '    logger.go:42: 09:32:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.335143 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.335147 | controller |     - '    logger.go:42: 09:32:00 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.335152 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.335156 | controller |       found'
2026-03-14 09:44:27.335162 | controller |     - '    logger.go:42: 09:32:01 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.335166 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.335170 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.335174 | controller |     - '        '
2026-03-14 09:44:27.335179 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.335183 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.335187 | controller |     - '        '
2026-03-14 09:44:27.335191 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.335195 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.335200 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.335206 | controller |     - '          exit 1'
2026-03-14 09:44:27.335210 | controller |     - '        fi'
2026-03-14 09:44:27.335214 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.335218 | controller |     - '        '
2026-03-14 09:44:27.335223 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.335227 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.335231 | controller |       | base64 -d)'
2026-03-14 09:44:27.335235 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.335239 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.335244 | controller |     - '          exit 1'
2026-03-14 09:44:27.335248 | controller |     - '        fi'
2026-03-14 09:44:27.335252 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.335256 | controller |     - '        '
2026-03-14 09:44:27.335261 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.335265 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.335269 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.335273 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.335277 | controller |       --timeout=300s'
2026-03-14 09:44:27.335282 | controller |     - '        '
2026-03-14 09:44:27.335286 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.335308 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.335316 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.335320 | controller |     - '        '
2026-03-14 09:44:27.335335 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.335339 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.335343 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.335347 | controller |     - '        '
2026-03-14 09:44:27.335352 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.335356 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.335360 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.335364 | controller |       available"'
2026-03-14 09:44:27.335369 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.335373 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.335377 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.335381 | controller |     - '          exit 1'
2026-03-14 09:44:27.335385 | controller |     - '        fi'
2026-03-14 09:44:27.335390 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.335394 | controller |     - '        '
2026-03-14 09:44:27.335398 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.335402 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.335406 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.335410 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.335415 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.335419 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.335425 | controller |       -- \'
2026-03-14 09:44:27.335429 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.335433 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.335445 | controller |     - '    logger.go:42: 09:32:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.335449 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.335454 | controller |     - '    logger.go:42: 09:32:01 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.335460 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.335466 | controller |       found'
2026-03-14 09:44:27.335471 | controller |     - '    logger.go:42: 09:32:02 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.335476 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.335480 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.335484 | controller |     - '        '
2026-03-14 09:44:27.335488 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.335493 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.335497 | controller |     - '        '
2026-03-14 09:44:27.335501 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.335505 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.335512 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.335516 | controller |     - '          exit 1'
2026-03-14 09:44:27.335520 | controller |     - '        fi'
2026-03-14 09:44:27.335525 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.335529 | controller |     - '        '
2026-03-14 09:44:27.335533 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.335538 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.335544 | controller |       | base64 -d)'
2026-03-14 09:44:27.335550 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.335554 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.335558 | controller |     - '          exit 1'
2026-03-14 09:44:27.335563 | controller |     - '        fi'
2026-03-14 09:44:27.335567 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.335571 | controller |     - '        '
2026-03-14 09:44:27.335576 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.335580 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.335584 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.335588 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.335592 | controller |       --timeout=300s'
2026-03-14 09:44:27.335597 | controller |     - '        '
2026-03-14 09:44:27.335601 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.335605 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.335609 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.335614 | controller |     - '        '
2026-03-14 09:44:27.335619 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.335628 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.335632 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.335638 | controller |     - '        '
2026-03-14 09:44:27.335642 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.335647 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.335651 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.335655 | controller |       available"'
2026-03-14 09:44:27.335659 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.335663 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.335668 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.335672 | controller |     - '          exit 1'
2026-03-14 09:44:27.335676 | controller |     - '        fi'
2026-03-14 09:44:27.335680 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.335685 | controller |     - '        '
2026-03-14 09:44:27.335689 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.335693 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.335699 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.335705 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.335709 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.335713 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.335718 | controller |       -- \'
2026-03-14 09:44:27.335722 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.335726 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.335736 | controller |     - '    logger.go:42: 09:32:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.335741 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.335746 | controller |     - '    logger.go:42: 09:32:02 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.335750 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.335754 | controller |       found'
2026-03-14 09:44:27.335759 | controller |     - '    logger.go:42: 09:32:03 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.335763 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.335767 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.335772 | controller |     - '        '
2026-03-14 09:44:27.335780 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.335786 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.335790 | controller |     - '        '
2026-03-14 09:44:27.335794 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.335798 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.335803 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.335807 | controller |     - '          exit 1'
2026-03-14 09:44:27.335811 | controller |     - '        fi'
2026-03-14 09:44:27.335817 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.335821 | controller |     - '        '
2026-03-14 09:44:27.335826 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.335832 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.335838 | controller |       | base64 -d)'
2026-03-14 09:44:27.335844 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.335852 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.335858 | controller |     - '          exit 1'
2026-03-14 09:44:27.335864 | controller |     - '        fi'
2026-03-14 09:44:27.335870 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.335875 | controller |     - '        '
2026-03-14 09:44:27.335880 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.335884 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.335888 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.335892 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.335896 | controller |       --timeout=300s'
2026-03-14 09:44:27.335901 | controller |     - '        '
2026-03-14 09:44:27.335905 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.335909 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.335913 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.335917 | controller |     - '        '
2026-03-14 09:44:27.335922 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.335926 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.335930 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.335934 | controller |     - '        '
2026-03-14 09:44:27.335938 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.335943 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.335947 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.335951 | controller |       available"'
2026-03-14 09:44:27.335955 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.335959 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.335963 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.335968 | controller |     - '          exit 1'
2026-03-14 09:44:27.335972 | controller |     - '        fi'
2026-03-14 09:44:27.335976 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.335980 | controller |     - '        '
2026-03-14 09:44:27.335984 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.335988 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.335993 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.335997 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.336001 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.336005 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.336009 | controller |       -- \'
2026-03-14 09:44:27.336014 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.336018 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.336022 | controller |     - '    logger.go:42: 09:32:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.336033 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.336039 | controller |     - '    logger.go:42: 09:32:03 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.336043 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.336048 | controller |       found'
2026-03-14 09:44:27.336052 | controller |     - '    logger.go:42: 09:32:04 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.336056 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.336060 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.336065 | controller |     - '        '
2026-03-14 09:44:27.336069 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.336073 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.336077 | controller |     - '        '
2026-03-14 09:44:27.336081 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.336086 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.336090 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.336094 | controller |     - '          exit 1'
2026-03-14 09:44:27.336098 | controller |     - '        fi'
2026-03-14 09:44:27.336102 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.336107 | controller |     - '        '
2026-03-14 09:44:27.336111 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.336115 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.336119 | controller |       | base64 -d)'
2026-03-14 09:44:27.336123 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.336128 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.336132 | controller |     - '          exit 1'
2026-03-14 09:44:27.336136 | controller |     - '        fi'
2026-03-14 09:44:27.336140 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.336145 | controller |     - '        '
2026-03-14 09:44:27.336149 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.336153 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.336157 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.336161 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.336166 | controller |       --timeout=300s'
2026-03-14 09:44:27.336170 | controller |     - '        '
2026-03-14 09:44:27.336174 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.336178 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.336182 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.336186 | controller |     - '        '
2026-03-14 09:44:27.336191 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.336195 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.336199 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.336203 | controller |     - '        '
2026-03-14 09:44:27.336208 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.336212 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.336216 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.336221 | controller |       available"'
2026-03-14 09:44:27.336226 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.336234 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.336240 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.336245 | controller |     - '          exit 1'
2026-03-14 09:44:27.336251 | controller |     - '        fi'
2026-03-14 09:44:27.336257 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.336263 | controller |     - '        '
2026-03-14 09:44:27.336269 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.336275 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.336281 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.336290 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.336317 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.336324 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.336330 | controller |       -- \'
2026-03-14 09:44:27.336335 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.336341 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.336346 | controller |     - '    logger.go:42: 09:32:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.336352 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.336387 | controller |     - '    logger.go:42: 09:32:04 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.336396 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.336402 | controller |       found'
2026-03-14 09:44:27.336408 | controller |     - '    logger.go:42: 09:32:05 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.336414 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.336420 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.336426 | controller |     - '        '
2026-03-14 09:44:27.336431 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.336439 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.336443 | controller |     - '        '
2026-03-14 09:44:27.336447 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.336452 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.336456 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.336460 | controller |     - '          exit 1'
2026-03-14 09:44:27.336464 | controller |     - '        fi'
2026-03-14 09:44:27.336469 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.336473 | controller |     - '        '
2026-03-14 09:44:27.336479 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.336483 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.336487 | controller |       | base64 -d)'
2026-03-14 09:44:27.336491 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.336496 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.336500 | controller |     - '          exit 1'
2026-03-14 09:44:27.336504 | controller |     - '        fi'
2026-03-14 09:44:27.336508 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.336512 | controller |     - '        '
2026-03-14 09:44:27.336517 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.336521 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.336525 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.336531 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.336536 | controller |       --timeout=300s'
2026-03-14 09:44:27.336540 | controller |     - '        '
2026-03-14 09:44:27.336544 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.336548 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.336552 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.336557 | controller |     - '        '
2026-03-14 09:44:27.336561 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.336565 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.336569 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.336574 | controller |     - '        '
2026-03-14 09:44:27.336578 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.336582 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.336586 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.336590 | controller |       available"'
2026-03-14 09:44:27.336594 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.336599 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.336603 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.336607 | controller |     - '          exit 1'
2026-03-14 09:44:27.336611 | controller |     - '        fi'
2026-03-14 09:44:27.336615 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.336620 | controller |     - '        '
2026-03-14 09:44:27.336624 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.336628 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.336632 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.336637 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.336641 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.336645 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.336649 | controller |       -- \'
2026-03-14 09:44:27.336653 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.336658 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.336662 | controller |     - '    logger.go:42: 09:32:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.336666 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.336670 | controller |     - '    logger.go:42: 09:32:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.336682 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.336689 | controller |       found'
2026-03-14 09:44:27.336695 | controller |     - '    logger.go:42: 09:32:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.336699 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.336704 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.336708 | controller |     - '        '
2026-03-14 09:44:27.336712 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.336718 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.336722 | controller |     - '        '
2026-03-14 09:44:27.336726 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.336730 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.336735 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.336739 | controller |     - '          exit 1'
2026-03-14 09:44:27.336743 | controller |     - '        fi'
2026-03-14 09:44:27.336747 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.336751 | controller |     - '        '
2026-03-14 09:44:27.336756 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.336760 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.336764 | controller |       | base64 -d)'
2026-03-14 09:44:27.336768 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.336772 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.336777 | controller |     - '          exit 1'
2026-03-14 09:44:27.336781 | controller |     - '        fi'
2026-03-14 09:44:27.336785 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.336789 | controller |     - '        '
2026-03-14 09:44:27.336793 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.336797 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.336802 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.336806 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.336810 | controller |       --timeout=300s'
2026-03-14 09:44:27.336818 | controller |     - '        '
2026-03-14 09:44:27.336822 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.336826 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.336831 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.336835 | controller |     - '        '
2026-03-14 09:44:27.336839 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.336843 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.336848 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.336852 | controller |     - '        '
2026-03-14 09:44:27.336856 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.336860 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.336864 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.336869 | controller |       available"'
2026-03-14 09:44:27.336873 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.336877 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.336881 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.336885 | controller |     - '          exit 1'
2026-03-14 09:44:27.336890 | controller |     - '        fi'
2026-03-14 09:44:27.336896 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.336900 | controller |     - '        '
2026-03-14 09:44:27.336904 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.336908 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.336914 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.336918 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.336922 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.336926 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.336931 | controller |       -- \'
2026-03-14 09:44:27.336935 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.336939 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.336943 | controller |     - '    logger.go:42: 09:32:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.336947 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.336952 | controller |     - '    logger.go:42: 09:32:06 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.336962 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.336967 | controller |       found'
2026-03-14 09:44:27.336971 | controller |     - '    logger.go:42: 09:32:07 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.336975 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.336980 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.336984 | controller |     - '        '
2026-03-14 09:44:27.336988 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.336992 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.336996 | controller |     - '        '
2026-03-14 09:44:27.337002 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.337006 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.337011 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.337015 | controller |     - '          exit 1'
2026-03-14 09:44:27.337019 | controller |     - '        fi'
2026-03-14 09:44:27.337023 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.337028 | controller |     - '        '
2026-03-14 09:44:27.337032 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.337036 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.337041 | controller |       | base64 -d)'
2026-03-14 09:44:27.337045 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.337050 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.337055 | controller |     - '          exit 1'
2026-03-14 09:44:27.337059 | controller |     - '        fi'
2026-03-14 09:44:27.337063 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.337067 | controller |     - '        '
2026-03-14 09:44:27.337072 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.337076 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.337080 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.337084 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.337089 | controller |       --timeout=300s'
2026-03-14 09:44:27.337094 | controller |     - '        '
2026-03-14 09:44:27.337098 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.337102 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.337106 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.337112 | controller |     - '        '
2026-03-14 09:44:27.337116 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.337120 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.337125 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.337129 | controller |     - '        '
2026-03-14 09:44:27.337133 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.337137 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.337141 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.337146 | controller |       available"'
2026-03-14 09:44:27.337150 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.337154 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.337158 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.337163 | controller |     - '          exit 1'
2026-03-14 09:44:27.337167 | controller |     - '        fi'
2026-03-14 09:44:27.337171 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.337175 | controller |     - '        '
2026-03-14 09:44:27.337179 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.337184 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.337188 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.337192 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.337196 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.337200 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.337205 | controller |       -- \'
2026-03-14 09:44:27.337209 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.337213 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.337217 | controller |     - '    logger.go:42: 09:32:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.337222 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.337226 | controller |     - '    logger.go:42: 09:32:08 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.337230 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.337234 | controller |       found'
2026-03-14 09:44:27.337243 | controller |     - '    logger.go:42: 09:32:09 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.337248 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.337253 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.337257 | controller |     - '        '
2026-03-14 09:44:27.337262 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.337266 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.337270 | controller |     - '        '
2026-03-14 09:44:27.337274 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.337278 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.337283 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.337290 | controller |     - '          exit 1'
2026-03-14 09:44:27.337308 | controller |     - '        fi'
2026-03-14 09:44:27.337315 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.337320 | controller |     - '        '
2026-03-14 09:44:27.337324 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.337328 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.337332 | controller |       | base64 -d)'
2026-03-14 09:44:27.337336 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.337341 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.337345 | controller |     - '          exit 1'
2026-03-14 09:44:27.337349 | controller |     - '        fi'
2026-03-14 09:44:27.337353 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.337358 | controller |     - '        '
2026-03-14 09:44:27.337362 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.337366 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.337370 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.337374 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.337378 | controller |       --timeout=300s'
2026-03-14 09:44:27.337386 | controller |     - '        '
2026-03-14 09:44:27.337392 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.337398 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.337403 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.337409 | controller |     - '        '
2026-03-14 09:44:27.337415 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.337420 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.337426 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.337431 | controller |     - '        '
2026-03-14 09:44:27.337437 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.337442 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.337447 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.337453 | controller |       available"'
2026-03-14 09:44:27.337457 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.337461 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.337465 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.337469 | controller |     - '          exit 1'
2026-03-14 09:44:27.337476 | controller |     - '        fi'
2026-03-14 09:44:27.337480 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.337484 | controller |     - '        '
2026-03-14 09:44:27.337488 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.337492 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.337497 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.337501 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.337505 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.337509 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.337515 | controller |       -- \'
2026-03-14 09:44:27.337519 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.337523 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.337528 | controller |     - '    logger.go:42: 09:32:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.337542 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.337547 | controller |     - '    logger.go:42: 09:32:09 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.337551 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.337555 | controller |       found'
2026-03-14 09:44:27.337567 | controller |     - '    logger.go:42: 09:32:10 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.337571 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.337575 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.337580 | controller |     - '        '
2026-03-14 09:44:27.337584 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.337588 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.337592 | controller |     - '        '
2026-03-14 09:44:27.337596 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.337601 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.337605 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.337609 | controller |     - '          exit 1'
2026-03-14 09:44:27.337613 | controller |     - '        fi'
2026-03-14 09:44:27.337618 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.337624 | controller |     - '        '
2026-03-14 09:44:27.337629 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.337633 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.337637 | controller |       | base64 -d)'
2026-03-14 09:44:27.337641 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.337645 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.337650 | controller |     - '          exit 1'
2026-03-14 09:44:27.337654 | controller |     - '        fi'
2026-03-14 09:44:27.337658 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.337662 | controller |     - '        '
2026-03-14 09:44:27.337667 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.337671 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.337675 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.337679 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.337683 | controller |       --timeout=300s'
2026-03-14 09:44:27.337688 | controller |     - '        '
2026-03-14 09:44:27.337692 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.337696 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.337700 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.337704 | controller |     - '        '
2026-03-14 09:44:27.337709 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.337713 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.337717 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.337723 | controller |     - '        '
2026-03-14 09:44:27.337727 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.337731 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.337735 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.337740 | controller |       available"'
2026-03-14 09:44:27.337744 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.337749 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.337754 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.337758 | controller |     - '          exit 1'
2026-03-14 09:44:27.337762 | controller |     - '        fi'
2026-03-14 09:44:27.337766 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.337770 | controller |     - '        '
2026-03-14 09:44:27.337775 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.337779 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.337783 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.337787 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.337791 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.337796 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.337800 | controller |       -- \'
2026-03-14 09:44:27.337804 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.337808 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.337812 | controller |     - '    logger.go:42: 09:32:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.337817 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.337821 | controller |     - '    logger.go:42: 09:32:10 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.337825 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.337829 | controller |       found'
2026-03-14 09:44:27.337834 | controller |     - '    logger.go:42: 09:32:11 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.337839 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.337851 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.337857 | controller |     - '        '
2026-03-14 09:44:27.337863 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.337869 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.337875 | controller |     - '        '
2026-03-14 09:44:27.337880 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.337886 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.337892 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.337898 | controller |     - '          exit 1'
2026-03-14 09:44:27.337904 | controller |     - '        fi'
2026-03-14 09:44:27.337910 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.337915 | controller |     - '        '
2026-03-14 09:44:27.337919 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.337923 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.337927 | controller |       | base64 -d)'
2026-03-14 09:44:27.337931 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.337936 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.337942 | controller |     - '          exit 1'
2026-03-14 09:44:27.337947 | controller |     - '        fi'
2026-03-14 09:44:27.337951 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.337955 | controller |     - '        '
2026-03-14 09:44:27.337959 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.337963 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.337968 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.337972 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.337976 | controller |       --timeout=300s'
2026-03-14 09:44:27.337980 | controller |     - '        '
2026-03-14 09:44:27.338010 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.338015 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.338019 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.338024 | controller |     - '        '
2026-03-14 09:44:27.338029 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.338034 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.338040 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.338046 | controller |     - '        '
2026-03-14 09:44:27.338052 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.338058 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.338064 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.338070 | controller |       available"'
2026-03-14 09:44:27.338076 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.338082 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.338087 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.338093 | controller |     - '          exit 1'
2026-03-14 09:44:27.338099 | controller |     - '        fi'
2026-03-14 09:44:27.338105 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.338110 | controller |     - '        '
2026-03-14 09:44:27.338116 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.338122 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.338128 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.338133 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.338137 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.338141 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.338146 | controller |       -- \'
2026-03-14 09:44:27.338150 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.338154 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.338159 | controller |     - '    logger.go:42: 09:32:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.338177 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.338183 | controller |     - '    logger.go:42: 09:32:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.338193 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.338199 | controller |       found'
2026-03-14 09:44:27.338205 | controller |     - '    logger.go:42: 09:32:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.338210 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.338216 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.338222 | controller |     - '        '
2026-03-14 09:44:27.338236 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.338242 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.338246 | controller |     - '        '
2026-03-14 09:44:27.338250 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.338255 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.338259 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.338263 | controller |     - '          exit 1'
2026-03-14 09:44:27.338267 | controller |     - '        fi'
2026-03-14 09:44:27.338271 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.338276 | controller |     - '        '
2026-03-14 09:44:27.338280 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.338284 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.338288 | controller |       | base64 -d)'
2026-03-14 09:44:27.338311 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.338315 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.338319 | controller |     - '          exit 1'
2026-03-14 09:44:27.338324 | controller |     - '        fi'
2026-03-14 09:44:27.338328 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.338332 | controller |     - '        '
2026-03-14 09:44:27.338336 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.338341 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.338348 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.338352 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.338357 | controller |       --timeout=300s'
2026-03-14 09:44:27.338361 | controller |     - '        '
2026-03-14 09:44:27.338365 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.338369 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.338373 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.338378 | controller |     - '        '
2026-03-14 09:44:27.338382 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.338386 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.338390 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.338394 | controller |     - '        '
2026-03-14 09:44:27.338399 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.338403 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.338407 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.338411 | controller |       available"'
2026-03-14 09:44:27.338416 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.338420 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.338427 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.338431 | controller |     - '          exit 1'
2026-03-14 09:44:27.338435 | controller |     - '        fi'
2026-03-14 09:44:27.338440 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.338444 | controller |     - '        '
2026-03-14 09:44:27.338448 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.338452 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.338457 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.338461 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.338465 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.338469 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.338475 | controller |       -- \'
2026-03-14 09:44:27.338479 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.338483 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.338489 | controller |     - '    logger.go:42: 09:32:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.338493 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.338497 | controller |     - '    logger.go:42: 09:32:12 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.338502 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.338506 | controller |       found'
2026-03-14 09:44:27.338510 | controller |     - '    logger.go:42: 09:32:13 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.338514 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.338518 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.338523 | controller |     - '        '
2026-03-14 09:44:27.338533 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.338538 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.338542 | controller |     - '        '
2026-03-14 09:44:27.338546 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.338551 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.338555 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.338559 | controller |     - '          exit 1'
2026-03-14 09:44:27.338563 | controller |     - '        fi'
2026-03-14 09:44:27.338568 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.338572 | controller |     - '        '
2026-03-14 09:44:27.338576 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.338580 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.338584 | controller |       | base64 -d)'
2026-03-14 09:44:27.338589 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.338594 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.338598 | controller |     - '          exit 1'
2026-03-14 09:44:27.338602 | controller |     - '        fi'
2026-03-14 09:44:27.338607 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.338611 | controller |     - '        '
2026-03-14 09:44:27.338615 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.338619 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.338623 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.338630 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.338635 | controller |       --timeout=300s'
2026-03-14 09:44:27.338641 | controller |     - '        '
2026-03-14 09:44:27.338647 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.338653 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.338658 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.338664 | controller |     - '        '
2026-03-14 09:44:27.338669 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.338675 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.338681 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.338686 | controller |     - '        '
2026-03-14 09:44:27.338692 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.338697 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.338703 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.338708 | controller |       available"'
2026-03-14 09:44:27.338714 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.338720 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.338725 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.338731 | controller |     - '          exit 1'
2026-03-14 09:44:27.338737 | controller |     - '        fi'
2026-03-14 09:44:27.338742 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.338748 | controller |     - '        '
2026-03-14 09:44:27.338754 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.338760 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.338766 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.338771 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.338777 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.338783 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.338788 | controller |       -- \'
2026-03-14 09:44:27.338794 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.338800 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.338806 | controller |     - '    logger.go:42: 09:32:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.338811 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.338817 | controller |     - '    logger.go:42: 09:32:13 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.338823 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.338828 | controller |       found'
2026-03-14 09:44:27.338834 | controller |     - '    logger.go:42: 09:32:14 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.338840 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.338845 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.338851 | controller |     - '        '
2026-03-14 09:44:27.338857 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.338872 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.338879 | controller |     - '        '
2026-03-14 09:44:27.338884 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.338890 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.338896 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.338901 | controller |     - '          exit 1'
2026-03-14 09:44:27.338907 | controller |     - '        fi'
2026-03-14 09:44:27.338912 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.338919 | controller |     - '        '
2026-03-14 09:44:27.338924 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.338930 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.338936 | controller |       | base64 -d)'
2026-03-14 09:44:27.338942 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.338947 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.338951 | controller |     - '          exit 1'
2026-03-14 09:44:27.338956 | controller |     - '        fi'
2026-03-14 09:44:27.338960 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.338964 | controller |     - '        '
2026-03-14 09:44:27.338968 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.338973 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.338977 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.338983 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.338988 | controller |       --timeout=300s'
2026-03-14 09:44:27.338992 | controller |     - '        '
2026-03-14 09:44:27.338996 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.339000 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.339005 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.339009 | controller |     - '        '
2026-03-14 09:44:27.339013 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.339018 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.339022 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.339026 | controller |     - '        '
2026-03-14 09:44:27.339030 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.339034 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.339039 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.339043 | controller |       available"'
2026-03-14 09:44:27.339047 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.339051 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.339055 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.339060 | controller |     - '          exit 1'
2026-03-14 09:44:27.339064 | controller |     - '        fi'
2026-03-14 09:44:27.339068 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.339072 | controller |     - '        '
2026-03-14 09:44:27.339076 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.339081 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.339086 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.339091 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.339095 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.339099 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.339103 | controller |       -- \'
2026-03-14 09:44:27.339107 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.339112 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.339116 | controller |     - '    logger.go:42: 09:32:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.339120 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.339124 | controller |     - '    logger.go:42: 09:32:14 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.339128 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.339133 | controller |       found'
2026-03-14 09:44:27.339137 | controller |     - '    logger.go:42: 09:32:15 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.339141 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.339145 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.339150 | controller |     - '        '
2026-03-14 09:44:27.339154 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.339165 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.339169 | controller |     - '        '
2026-03-14 09:44:27.339173 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.339178 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.339182 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.339186 | controller |     - '          exit 1'
2026-03-14 09:44:27.339190 | controller |     - '        fi'
2026-03-14 09:44:27.339195 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.339199 | controller |     - '        '
2026-03-14 09:44:27.339205 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.339209 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.339213 | controller |       | base64 -d)'
2026-03-14 09:44:27.339219 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.339223 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.339227 | controller |     - '          exit 1'
2026-03-14 09:44:27.339231 | controller |     - '        fi'
2026-03-14 09:44:27.339236 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.339240 | controller |     - '        '
2026-03-14 09:44:27.339244 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.339248 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.339253 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.339257 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.339261 | controller |       --timeout=300s'
2026-03-14 09:44:27.339265 | controller |     - '        '
2026-03-14 09:44:27.339269 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.339274 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.339278 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.339284 | controller |     - '        '
2026-03-14 09:44:27.339301 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.339308 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.339313 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.339317 | controller |     - '        '
2026-03-14 09:44:27.339321 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.339325 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.339330 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.339334 | controller |       available"'
2026-03-14 09:44:27.339338 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.339342 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.339346 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.339353 | controller |     - '          exit 1'
2026-03-14 09:44:27.339357 | controller |     - '        fi'
2026-03-14 09:44:27.339362 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.339366 | controller |     - '        '
2026-03-14 09:44:27.339370 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.339374 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.339378 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.339392 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.339397 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.339401 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.339405 | controller |       -- \'
2026-03-14 09:44:27.339410 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.339414 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.339418 | controller |     - '    logger.go:42: 09:32:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.339422 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.339426 | controller |     - '    logger.go:42: 09:32:16 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.339431 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.339435 | controller |       found'
2026-03-14 09:44:27.339439 | controller |     - '    logger.go:42: 09:32:17 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.339443 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.339450 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.339454 | controller |     - '        '
2026-03-14 09:44:27.339459 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.339463 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.339467 | controller |     - '        '
2026-03-14 09:44:27.339478 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.339482 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.339487 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.339491 | controller |     - '          exit 1'
2026-03-14 09:44:27.339495 | controller |     - '        fi'
2026-03-14 09:44:27.339499 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.339505 | controller |     - '        '
2026-03-14 09:44:27.339510 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.339514 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.339518 | controller |       | base64 -d)'
2026-03-14 09:44:27.339522 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.339526 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.339531 | controller |     - '          exit 1'
2026-03-14 09:44:27.339535 | controller |     - '        fi'
2026-03-14 09:44:27.339539 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.339543 | controller |     - '        '
2026-03-14 09:44:27.339547 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.339552 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.339556 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.339560 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.339564 | controller |       --timeout=300s'
2026-03-14 09:44:27.339568 | controller |     - '        '
2026-03-14 09:44:27.339573 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.339577 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.339581 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.339585 | controller |     - '        '
2026-03-14 09:44:27.339589 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.339594 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.339601 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.339605 | controller |     - '        '
2026-03-14 09:44:27.339611 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.339615 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.339619 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.339624 | controller |       available"'
2026-03-14 09:44:27.339628 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.339632 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.339636 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.339640 | controller |     - '          exit 1'
2026-03-14 09:44:27.339644 | controller |     - '        fi'
2026-03-14 09:44:27.339649 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.339653 | controller |     - '        '
2026-03-14 09:44:27.339657 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.339661 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.339666 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.339670 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.339674 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.339678 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.339682 | controller |       -- \'
2026-03-14 09:44:27.339688 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.339692 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.339697 | controller |     - '    logger.go:42: 09:32:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.339701 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.339705 | controller |     - '    logger.go:42: 09:32:17 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.339709 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.339713 | controller |       found'
2026-03-14 09:44:27.339718 | controller |     - '    logger.go:42: 09:32:18 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.339722 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.339726 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.339730 | controller |     - '        '
2026-03-14 09:44:27.339734 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.339739 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.339743 | controller |     - '        '
2026-03-14 09:44:27.339753 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.339757 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.339762 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.339766 | controller |     - '          exit 1'
2026-03-14 09:44:27.339770 | controller |     - '        fi'
2026-03-14 09:44:27.339774 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.339779 | controller |     - '        '
2026-03-14 09:44:27.339783 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.339787 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.339791 | controller |       | base64 -d)'
2026-03-14 09:44:27.339795 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.339800 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.339804 | controller |     - '          exit 1'
2026-03-14 09:44:27.339808 | controller |     - '        fi'
2026-03-14 09:44:27.339812 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.339817 | controller |     - '        '
2026-03-14 09:44:27.339821 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.339825 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.339829 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.339833 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.339838 | controller |       --timeout=300s'
2026-03-14 09:44:27.339842 | controller |     - '        '
2026-03-14 09:44:27.339846 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.339850 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.339854 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.339859 | controller |     - '        '
2026-03-14 09:44:27.339864 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.339869 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.339873 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.339877 | controller |     - '        '
2026-03-14 09:44:27.339881 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.339892 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.339896 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.339900 | controller |       available"'
2026-03-14 09:44:27.339904 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.339908 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.339913 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.339917 | controller |     - '          exit 1'
2026-03-14 09:44:27.339921 | controller |     - '        fi'
2026-03-14 09:44:27.339925 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.339929 | controller |     - '        '
2026-03-14 09:44:27.339934 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.339938 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.339942 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.339946 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.339951 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.339955 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.339959 | controller |       -- \'
2026-03-14 09:44:27.339963 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.339967 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.339971 | controller |     - '    logger.go:42: 09:32:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.339976 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.339980 | controller |     - '    logger.go:42: 09:32:18 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.339984 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.339988 | controller |       found'
2026-03-14 09:44:27.339992 | controller |     - '    logger.go:42: 09:32:19 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.339997 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.340001 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.340005 | controller |     - '        '
2026-03-14 09:44:27.340009 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.340013 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.340018 | controller |     - '        '
2026-03-14 09:44:27.340022 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.340031 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.340035 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.340040 | controller |     - '          exit 1'
2026-03-14 09:44:27.340045 | controller |     - '        fi'
2026-03-14 09:44:27.340051 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.340057 | controller |     - '        '
2026-03-14 09:44:27.340063 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.340069 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.340075 | controller |       | base64 -d)'
2026-03-14 09:44:27.340081 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.340087 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.340093 | controller |     - '          exit 1'
2026-03-14 09:44:27.340101 | controller |     - '        fi'
2026-03-14 09:44:27.340107 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.340113 | controller |     - '        '
2026-03-14 09:44:27.340119 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.340124 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.340128 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.340132 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.340136 | controller |       --timeout=300s'
2026-03-14 09:44:27.340141 | controller |     - '        '
2026-03-14 09:44:27.340145 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.340149 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.340153 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.340158 | controller |     - '        '
2026-03-14 09:44:27.340164 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.340170 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.340176 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.340182 | controller |     - '        '
2026-03-14 09:44:27.340188 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.340194 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.340203 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.340209 | controller |       available"'
2026-03-14 09:44:27.340215 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.340221 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.340227 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.340232 | controller |     - '          exit 1'
2026-03-14 09:44:27.340237 | controller |     - '        fi'
2026-03-14 09:44:27.340241 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.340245 | controller |     - '        '
2026-03-14 09:44:27.340249 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.340253 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.340258 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.340264 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.340270 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.340276 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.340282 | controller |       -- \'
2026-03-14 09:44:27.340303 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.340314 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.340320 | controller |     - '    logger.go:42: 09:32:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.340326 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.340332 | controller |     - '    logger.go:42: 09:32:19 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.340338 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.340347 | controller |       found'
2026-03-14 09:44:27.340353 | controller |     - '    logger.go:42: 09:32:20 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.340359 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.340365 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.340371 | controller |     - '        '
2026-03-14 09:44:27.340379 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.340385 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.340390 | controller |     - '        '
2026-03-14 09:44:27.340396 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.340402 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.340417 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.340425 | controller |     - '          exit 1'
2026-03-14 09:44:27.340431 | controller |     - '        fi'
2026-03-14 09:44:27.340436 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.340442 | controller |     - '        '
2026-03-14 09:44:27.340448 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.340454 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.340460 | controller |       | base64 -d)'
2026-03-14 09:44:27.340465 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.340471 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.340477 | controller |     - '          exit 1'
2026-03-14 09:44:27.340483 | controller |     - '        fi'
2026-03-14 09:44:27.340489 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.340494 | controller |     - '        '
2026-03-14 09:44:27.340500 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.340506 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.340512 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.340518 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.340524 | controller |       --timeout=300s'
2026-03-14 09:44:27.340529 | controller |     - '        '
2026-03-14 09:44:27.340535 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.340541 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.340547 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.340553 | controller |     - '        '
2026-03-14 09:44:27.340559 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.340564 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.340570 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.340576 | controller |     - '        '
2026-03-14 09:44:27.340581 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.340586 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.340591 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.340595 | controller |       available"'
2026-03-14 09:44:27.340599 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.340606 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.340610 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.340617 | controller |     - '          exit 1'
2026-03-14 09:44:27.340621 | controller |     - '        fi'
2026-03-14 09:44:27.340625 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.340629 | controller |     - '        '
2026-03-14 09:44:27.340634 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.340640 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.340645 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.340651 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.340657 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.340662 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.340668 | controller |       -- \'
2026-03-14 09:44:27.340676 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.340682 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.340689 | controller |     - '    logger.go:42: 09:32:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.340695 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.340701 | controller |     - '    logger.go:42: 09:32:20 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.340707 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.340713 | controller |       found'
2026-03-14 09:44:27.340719 | controller |     - '    logger.go:42: 09:32:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.340724 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.340730 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.340736 | controller |     - '        '
2026-03-14 09:44:27.340742 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.340746 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.340751 | controller |     - '        '
2026-03-14 09:44:27.340755 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.340759 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.340763 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.340768 | controller |     - '          exit 1'
2026-03-14 09:44:27.340780 | controller |     - '        fi'
2026-03-14 09:44:27.340785 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.340789 | controller |     - '        '
2026-03-14 09:44:27.340793 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.340798 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.340802 | controller |       | base64 -d)'
2026-03-14 09:44:27.340806 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.340810 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.340815 | controller |     - '          exit 1'
2026-03-14 09:44:27.340819 | controller |     - '        fi'
2026-03-14 09:44:27.340823 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.340827 | controller |     - '        '
2026-03-14 09:44:27.340832 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.340836 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.340840 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.340844 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.340850 | controller |       --timeout=300s'
2026-03-14 09:44:27.340855 | controller |     - '        '
2026-03-14 09:44:27.340859 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.340863 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.340867 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.340872 | controller |     - '        '
2026-03-14 09:44:27.340876 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.340880 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.340884 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.340888 | controller |     - '        '
2026-03-14 09:44:27.340892 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.340897 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.340901 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.340905 | controller |       available"'
2026-03-14 09:44:27.340909 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.340915 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.340919 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.340923 | controller |     - '          exit 1'
2026-03-14 09:44:27.340928 | controller |     - '        fi'
2026-03-14 09:44:27.340932 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.340936 | controller |     - '        '
2026-03-14 09:44:27.340940 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.340944 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.340949 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.340953 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.340957 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.340961 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.340965 | controller |       -- \'
2026-03-14 09:44:27.340970 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.340974 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.340978 | controller |     - '    logger.go:42: 09:32:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.340982 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.340987 | controller |     - '    logger.go:42: 09:32:21 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.340991 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.340995 | controller |       found'
2026-03-14 09:44:27.340999 | controller |     - '    logger.go:42: 09:32:22 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.341003 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.341008 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.341012 | controller |     - '        '
2026-03-14 09:44:27.341016 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.341020 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.341026 | controller |     - '        '
2026-03-14 09:44:27.341030 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.341035 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.341039 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.341043 | controller |     - '          exit 1'
2026-03-14 09:44:27.341047 | controller |     - '        fi'
2026-03-14 09:44:27.341057 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.341061 | controller |     - '        '
2026-03-14 09:44:27.341066 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.341070 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.341074 | controller |       | base64 -d)'
2026-03-14 09:44:27.341078 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.341082 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.341087 | controller |     - '          exit 1'
2026-03-14 09:44:27.341091 | controller |     - '        fi'
2026-03-14 09:44:27.341095 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.341099 | controller |     - '        '
2026-03-14 09:44:27.341103 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.341108 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.341112 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.341116 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.341120 | controller |       --timeout=300s'
2026-03-14 09:44:27.341125 | controller |     - '        '
2026-03-14 09:44:27.341129 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.341133 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.341137 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.341141 | controller |     - '        '
2026-03-14 09:44:27.341146 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.341150 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.341154 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.341158 | controller |     - '        '
2026-03-14 09:44:27.341162 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.341167 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.341171 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.341175 | controller |       available"'
2026-03-14 09:44:27.341179 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.341183 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.341188 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.341192 | controller |     - '          exit 1'
2026-03-14 09:44:27.341196 | controller |     - '        fi'
2026-03-14 09:44:27.341200 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.341205 | controller |     - '        '
2026-03-14 09:44:27.341209 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.341213 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.341217 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.341224 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.341228 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.341233 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.341237 | controller |       -- \'
2026-03-14 09:44:27.341241 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.341245 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.341250 | controller |     - '    logger.go:42: 09:32:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.341254 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.341258 | controller |     - '    logger.go:42: 09:32:22 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.341262 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.341266 | controller |       found'
2026-03-14 09:44:27.341271 | controller |     - '    logger.go:42: 09:32:23 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.341275 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.341279 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.341283 | controller |     - '        '
2026-03-14 09:44:27.341290 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.341309 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.341314 | controller |     - '        '
2026-03-14 09:44:27.341318 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.341322 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.341326 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.341330 | controller |     - '          exit 1'
2026-03-14 09:44:27.341335 | controller |     - '        fi'
2026-03-14 09:44:27.341339 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.341343 | controller |     - '        '
2026-03-14 09:44:27.341353 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.341360 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.341366 | controller |       | base64 -d)'
2026-03-14 09:44:27.341371 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.341378 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.341382 | controller |     - '          exit 1'
2026-03-14 09:44:27.341387 | controller |     - '        fi'
2026-03-14 09:44:27.341391 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.341395 | controller |     - '        '
2026-03-14 09:44:27.341399 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.341404 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.341408 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.341412 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.341416 | controller |       --timeout=300s'
2026-03-14 09:44:27.341420 | controller |     - '        '
2026-03-14 09:44:27.341425 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.341429 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.341433 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.341439 | controller |     - '        '
2026-03-14 09:44:27.341443 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.341448 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.341452 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.341456 | controller |     - '        '
2026-03-14 09:44:27.341462 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.341467 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.341471 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.341475 | controller |       available"'
2026-03-14 09:44:27.341479 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.341484 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.341488 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.341492 | controller |     - '          exit 1'
2026-03-14 09:44:27.341497 | controller |     - '        fi'
2026-03-14 09:44:27.341501 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.341505 | controller |     - '        '
2026-03-14 09:44:27.341510 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.341515 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.341519 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.341523 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.341528 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.341532 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.341536 | controller |       -- \'
2026-03-14 09:44:27.341540 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.341544 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.341549 | controller |     - '    logger.go:42: 09:32:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.341553 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.341557 | controller |     - '    logger.go:42: 09:32:24 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.341561 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.341565 | controller |       found'
2026-03-14 09:44:27.341570 | controller |     - '    logger.go:42: 09:32:25 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.341574 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.341578 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.341582 | controller |     - '        '
2026-03-14 09:44:27.341587 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.341591 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.341595 | controller |     - '        '
2026-03-14 09:44:27.341599 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.341603 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.341608 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.341612 | controller |     - '          exit 1'
2026-03-14 09:44:27.341616 | controller |     - '        fi'
2026-03-14 09:44:27.341620 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.341624 | controller |     - '        '
2026-03-14 09:44:27.341630 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.341641 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.341645 | controller |       | base64 -d)'
2026-03-14 09:44:27.341650 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.341654 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.341658 | controller |     - '          exit 1'
2026-03-14 09:44:27.341662 | controller |     - '        fi'
2026-03-14 09:44:27.341667 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.341671 | controller |     - '        '
2026-03-14 09:44:27.341675 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.341679 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.341684 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.341688 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.341692 | controller |       --timeout=300s'
2026-03-14 09:44:27.341698 | controller |     - '        '
2026-03-14 09:44:27.341702 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.341706 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.341711 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.341715 | controller |     - '        '
2026-03-14 09:44:27.341719 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.341723 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.341728 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.341732 | controller |     - '        '
2026-03-14 09:44:27.341736 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.341740 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.341744 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.341749 | controller |       available"'
2026-03-14 09:44:27.341753 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.341757 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.341761 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.341765 | controller |     - '          exit 1'
2026-03-14 09:44:27.341770 | controller |     - '        fi'
2026-03-14 09:44:27.341774 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.341778 | controller |     - '        '
2026-03-14 09:44:27.341782 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.341787 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.341791 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.341795 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.341799 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.341804 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.341808 | controller |       -- \'
2026-03-14 09:44:27.341812 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.341819 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.341823 | controller |     - '    logger.go:42: 09:32:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.341827 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.341833 | controller |     - '    logger.go:42: 09:32:25 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.341838 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.341842 | controller |       found'
2026-03-14 09:44:27.341847 | controller |     - '    logger.go:42: 09:32:26 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.341851 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.341855 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.341859 | controller |     - '        '
2026-03-14 09:44:27.341864 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.341868 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.341872 | controller |     - '        '
2026-03-14 09:44:27.341876 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.341880 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.341885 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.341889 | controller |     - '          exit 1'
2026-03-14 09:44:27.341893 | controller |     - '        fi'
2026-03-14 09:44:27.341897 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.341901 | controller |     - '        '
2026-03-14 09:44:27.341907 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.341917 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.341922 | controller |       | base64 -d)'
2026-03-14 09:44:27.341926 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.341931 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.341935 | controller |     - '          exit 1'
2026-03-14 09:44:27.341939 | controller |     - '        fi'
2026-03-14 09:44:27.341943 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.341947 | controller |     - '        '
2026-03-14 09:44:27.341951 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.341956 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.341960 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.341964 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.341969 | controller |       --timeout=300s'
2026-03-14 09:44:27.341973 | controller |     - '        '
2026-03-14 09:44:27.341977 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.341982 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.341986 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.341990 | controller |     - '        '
2026-03-14 09:44:27.342013 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.342019 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.342024 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.342029 | controller |     - '        '
2026-03-14 09:44:27.342034 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.342041 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.342046 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.342051 | controller |       available"'
2026-03-14 09:44:27.342056 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.342061 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.342066 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.342070 | controller |     - '          exit 1'
2026-03-14 09:44:27.342075 | controller |     - '        fi'
2026-03-14 09:44:27.342080 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.342085 | controller |     - '        '
2026-03-14 09:44:27.342090 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.342095 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.342100 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.342107 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.342112 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.342117 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.342122 | controller |       -- \'
2026-03-14 09:44:27.342127 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.342132 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.342136 | controller |     - '    logger.go:42: 09:32:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.342141 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.342146 | controller |     - '    logger.go:42: 09:32:26 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.342151 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.342156 | controller |       found'
2026-03-14 09:44:27.342161 | controller |     - '    logger.go:42: 09:32:27 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.342166 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.342171 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.342176 | controller |     - '        '
2026-03-14 09:44:27.342181 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.342186 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.342191 | controller |     - '        '
2026-03-14 09:44:27.342195 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.342200 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.342205 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.342210 | controller |     - '          exit 1'
2026-03-14 09:44:27.342215 | controller |     - '        fi'
2026-03-14 09:44:27.342220 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.342225 | controller |     - '        '
2026-03-14 09:44:27.342230 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.342235 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.342245 | controller |       | base64 -d)'
2026-03-14 09:44:27.342251 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.342255 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.342259 | controller |     - '          exit 1'
2026-03-14 09:44:27.342263 | controller |     - '        fi'
2026-03-14 09:44:27.342269 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.342274 | controller |     - '        '
2026-03-14 09:44:27.342278 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.342282 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.342286 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.342303 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.342308 | controller |       --timeout=300s'
2026-03-14 09:44:27.342313 | controller |     - '        '
2026-03-14 09:44:27.342317 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.342321 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.342325 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.342330 | controller |     - '        '
2026-03-14 09:44:27.342334 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.342338 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.342342 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.342347 | controller |     - '        '
2026-03-14 09:44:27.342351 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.342355 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.342359 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.342364 | controller |       available"'
2026-03-14 09:44:27.342368 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.342372 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.342376 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.342380 | controller |     - '          exit 1'
2026-03-14 09:44:27.342385 | controller |     - '        fi'
2026-03-14 09:44:27.342389 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.342393 | controller |     - '        '
2026-03-14 09:44:27.342397 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.342402 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.342406 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.342410 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.342414 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.342419 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.342423 | controller |       -- \'
2026-03-14 09:44:27.342427 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.342431 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.342435 | controller |     - '    logger.go:42: 09:32:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.342440 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.342444 | controller |     - '    logger.go:42: 09:32:27 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.342448 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.342454 | controller |       found'
2026-03-14 09:44:27.342458 | controller |     - '    logger.go:42: 09:32:28 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.342463 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.342467 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.342471 | controller |     - '        '
2026-03-14 09:44:27.342475 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.342479 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.342484 | controller |     - '        '
2026-03-14 09:44:27.342488 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.342492 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.342496 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.342500 | controller |     - '          exit 1'
2026-03-14 09:44:27.342505 | controller |     - '        fi'
2026-03-14 09:44:27.342509 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.342513 | controller |     - '        '
2026-03-14 09:44:27.342517 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.342521 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.342526 | controller |       | base64 -d)'
2026-03-14 09:44:27.342536 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.342541 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.342545 | controller |     - '          exit 1'
2026-03-14 09:44:27.342549 | controller |     - '        fi'
2026-03-14 09:44:27.342557 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.342561 | controller |     - '        '
2026-03-14 09:44:27.342565 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.342570 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.342574 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.342578 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.342582 | controller |       --timeout=300s'
2026-03-14 09:44:27.342586 | controller |     - '        '
2026-03-14 09:44:27.342591 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.342595 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.342599 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.342603 | controller |     - '        '
2026-03-14 09:44:27.342608 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.342612 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.342616 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.342620 | controller |     - '        '
2026-03-14 09:44:27.342625 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.342631 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.342637 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.342642 | controller |       available"'
2026-03-14 09:44:27.342648 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.342654 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.342660 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.342665 | controller |     - '          exit 1'
2026-03-14 09:44:27.342673 | controller |     - '        fi'
2026-03-14 09:44:27.342679 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.342685 | controller |     - '        '
2026-03-14 09:44:27.342691 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.342697 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.342703 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.342709 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.342715 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.342721 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.342726 | controller |       -- \'
2026-03-14 09:44:27.342732 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.342739 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.342744 | controller |     - '    logger.go:42: 09:32:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.342749 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.342753 | controller |     - '    logger.go:42: 09:32:28 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.342757 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.342761 | controller |       found'
2026-03-14 09:44:27.342766 | controller |     - '    logger.go:42: 09:32:29 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.342770 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.342774 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.342778 | controller |     - '        '
2026-03-14 09:44:27.342782 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.342787 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.342791 | controller |     - '        '
2026-03-14 09:44:27.342795 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.342799 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.342803 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.342808 | controller |     - '          exit 1'
2026-03-14 09:44:27.342812 | controller |     - '        fi'
2026-03-14 09:44:27.342816 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.342820 | controller |     - '        '
2026-03-14 09:44:27.342824 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.342829 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.342833 | controller |       | base64 -d)'
2026-03-14 09:44:27.342837 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.342848 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.342853 | controller |     - '          exit 1'
2026-03-14 09:44:27.342857 | controller |     - '        fi'
2026-03-14 09:44:27.342861 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.342866 | controller |     - '        '
2026-03-14 09:44:27.342870 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.342874 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.342878 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.342883 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.342889 | controller |       --timeout=300s'
2026-03-14 09:44:27.342893 | controller |     - '        '
2026-03-14 09:44:27.342897 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.342902 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.342907 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.342912 | controller |     - '        '
2026-03-14 09:44:27.342916 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.342921 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.342926 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.342931 | controller |     - '        '
2026-03-14 09:44:27.342935 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.342940 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.342945 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.342950 | controller |       available"'
2026-03-14 09:44:27.342955 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.342959 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.342963 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.342967 | controller |     - '          exit 1'
2026-03-14 09:44:27.342971 | controller |     - '        fi'
2026-03-14 09:44:27.342975 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.342980 | controller |     - '        '
2026-03-14 09:44:27.342984 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.342988 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.342992 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.342996 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.343001 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.343005 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.343009 | controller |       -- \'
2026-03-14 09:44:27.343013 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.343017 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.343022 | controller |     - '    logger.go:42: 09:32:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.343026 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.343030 | controller |     - '    logger.go:42: 09:32:29 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.343034 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.343038 | controller |       found'
2026-03-14 09:44:27.343043 | controller |     - '    logger.go:42: 09:32:30 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.343047 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.343051 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.343055 | controller |     - '        '
2026-03-14 09:44:27.343059 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.343064 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.343068 | controller |     - '        '
2026-03-14 09:44:27.343075 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.343080 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.343086 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.343090 | controller |     - '          exit 1'
2026-03-14 09:44:27.343094 | controller |     - '        fi'
2026-03-14 09:44:27.343099 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.343103 | controller |     - '        '
2026-03-14 09:44:27.343107 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.343111 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.343116 | controller |       | base64 -d)'
2026-03-14 09:44:27.343120 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.343130 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.343135 | controller |     - '          exit 1'
2026-03-14 09:44:27.343139 | controller |     - '        fi'
2026-03-14 09:44:27.343143 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.343148 | controller |     - '        '
2026-03-14 09:44:27.343152 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.343156 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.343161 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.343165 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.343169 | controller |       --timeout=300s'
2026-03-14 09:44:27.343173 | controller |     - '        '
2026-03-14 09:44:27.343178 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.343182 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.343186 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.343190 | controller |     - '        '
2026-03-14 09:44:27.343194 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.343199 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.343203 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.343207 | controller |     - '        '
2026-03-14 09:44:27.343211 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.343216 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.343220 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.343224 | controller |       available"'
2026-03-14 09:44:27.343228 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.343233 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.343240 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.343244 | controller |     - '          exit 1'
2026-03-14 09:44:27.343248 | controller |     - '        fi'
2026-03-14 09:44:27.343252 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.343257 | controller |     - '        '
2026-03-14 09:44:27.343261 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.343265 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.343269 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.343275 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.343280 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.343284 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.343302 | controller |       -- \'
2026-03-14 09:44:27.343309 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.343314 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.343318 | controller |     - '    logger.go:42: 09:32:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.343323 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.343329 | controller |     - '    logger.go:42: 09:32:30 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.343333 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.343337 | controller |       found'
2026-03-14 09:44:27.343342 | controller |     - '    logger.go:42: 09:32:31 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.343346 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.343351 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.343355 | controller |     - '        '
2026-03-14 09:44:27.343359 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.343363 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.343368 | controller |     - '        '
2026-03-14 09:44:27.343372 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.343376 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.343380 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.343384 | controller |     - '          exit 1'
2026-03-14 09:44:27.343389 | controller |     - '        fi'
2026-03-14 09:44:27.343393 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.343397 | controller |     - '        '
2026-03-14 09:44:27.343401 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.343406 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.343410 | controller |       | base64 -d)'
2026-03-14 09:44:27.343414 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.343418 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.343423 | controller |     - '          exit 1'
2026-03-14 09:44:27.343433 | controller |     - '        fi'
2026-03-14 09:44:27.343438 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.343442 | controller |     - '        '
2026-03-14 09:44:27.343446 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.343450 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.343455 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.343459 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.343463 | controller |       --timeout=300s'
2026-03-14 09:44:27.343467 | controller |     - '        '
2026-03-14 09:44:27.343471 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.343476 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.343480 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.343484 | controller |     - '        '
2026-03-14 09:44:27.343488 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.343494 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.343499 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.343503 | controller |     - '        '
2026-03-14 09:44:27.343507 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.343511 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.343516 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.343520 | controller |       available"'
2026-03-14 09:44:27.343524 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.343528 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.343532 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.343537 | controller |     - '          exit 1'
2026-03-14 09:44:27.343541 | controller |     - '        fi'
2026-03-14 09:44:27.343545 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.343549 | controller |     - '        '
2026-03-14 09:44:27.343554 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.343558 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.343562 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.343566 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.343571 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.343575 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.343579 | controller |       -- \'
2026-03-14 09:44:27.343583 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.343587 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.343592 | controller |     - '    logger.go:42: 09:32:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.343596 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.343600 | controller |     - '    logger.go:42: 09:32:31 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.343604 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.343609 | controller |       found'
2026-03-14 09:44:27.343613 | controller |     - '    logger.go:42: 09:32:32 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.343617 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.343621 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.343625 | controller |     - '        '
2026-03-14 09:44:27.343630 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.343636 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.343642 | controller |     - '        '
2026-03-14 09:44:27.343647 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.343653 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.343659 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.343663 | controller |     - '          exit 1'
2026-03-14 09:44:27.343671 | controller |     - '        fi'
2026-03-14 09:44:27.343675 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.343679 | controller |     - '        '
2026-03-14 09:44:27.343683 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.343689 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.343693 | controller |       | base64 -d)'
2026-03-14 09:44:27.343698 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.343702 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.343706 | controller |     - '          exit 1'
2026-03-14 09:44:27.343710 | controller |     - '        fi'
2026-03-14 09:44:27.343721 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.343726 | controller |     - '        '
2026-03-14 09:44:27.343730 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.343734 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.343738 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.343742 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.343747 | controller |       --timeout=300s'
2026-03-14 09:44:27.343751 | controller |     - '        '
2026-03-14 09:44:27.343755 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.343759 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.343764 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.343768 | controller |     - '        '
2026-03-14 09:44:27.343772 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.343776 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.343780 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.343784 | controller |     - '        '
2026-03-14 09:44:27.343789 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.343793 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.343797 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.343801 | controller |       available"'
2026-03-14 09:44:27.343806 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.343810 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.343814 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.343818 | controller |     - '          exit 1'
2026-03-14 09:44:27.343822 | controller |     - '        fi'
2026-03-14 09:44:27.343827 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.343831 | controller |     - '        '
2026-03-14 09:44:27.343835 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.343839 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.343844 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.343848 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.343854 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.343858 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.343862 | controller |       -- \'
2026-03-14 09:44:27.343866 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.343873 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.343878 | controller |     - '    logger.go:42: 09:32:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.343882 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.343886 | controller |     - '    logger.go:42: 09:32:33 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.343892 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.343896 | controller |       found'
2026-03-14 09:44:27.343931 | controller |     - '    logger.go:42: 09:32:34 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.343938 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.343943 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.343947 | controller |     - '        '
2026-03-14 09:44:27.343951 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.343956 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.343960 | controller |     - '        '
2026-03-14 09:44:27.343964 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.343968 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.343972 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.343977 | controller |     - '          exit 1'
2026-03-14 09:44:27.343981 | controller |     - '        fi'
2026-03-14 09:44:27.343985 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.343989 | controller |     - '        '
2026-03-14 09:44:27.343994 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.343998 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.344002 | controller |       | base64 -d)'
2026-03-14 09:44:27.344006 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.344010 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.344015 | controller |     - '          exit 1'
2026-03-14 09:44:27.344019 | controller |     - '        fi'
2026-03-14 09:44:27.344023 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.344027 | controller |     - '        '
2026-03-14 09:44:27.344037 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.344042 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.344046 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.344051 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.344055 | controller |       --timeout=300s'
2026-03-14 09:44:27.344059 | controller |     - '        '
2026-03-14 09:44:27.344063 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.344068 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.344072 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.344076 | controller |     - '        '
2026-03-14 09:44:27.344083 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.344087 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.344092 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.344096 | controller |     - '        '
2026-03-14 09:44:27.344100 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.344104 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.344110 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.344115 | controller |       available"'
2026-03-14 09:44:27.344119 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.344123 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.344127 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.344132 | controller |     - '          exit 1'
2026-03-14 09:44:27.344136 | controller |     - '        fi'
2026-03-14 09:44:27.344140 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.344145 | controller |     - '        '
2026-03-14 09:44:27.344149 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.344153 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.344157 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.344162 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.344166 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.344170 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.344174 | controller |       -- \'
2026-03-14 09:44:27.344179 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.344183 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.344187 | controller |     - '    logger.go:42: 09:32:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.344191 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.344196 | controller |     - '    logger.go:42: 09:32:34 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.344200 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.344204 | controller |       found'
2026-03-14 09:44:27.344208 | controller |     - '    logger.go:42: 09:32:35 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.344213 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.344217 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.344221 | controller |     - '        '
2026-03-14 09:44:27.344226 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.344230 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.344234 | controller |     - '        '
2026-03-14 09:44:27.344238 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.344243 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.344247 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.344251 | controller |     - '          exit 1'
2026-03-14 09:44:27.344255 | controller |     - '        fi'
2026-03-14 09:44:27.344260 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.344264 | controller |     - '        '
2026-03-14 09:44:27.344270 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.344274 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.344278 | controller |       | base64 -d)'
2026-03-14 09:44:27.344282 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.344290 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.344308 | controller |     - '          exit 1'
2026-03-14 09:44:27.344313 | controller |     - '        fi'
2026-03-14 09:44:27.344317 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.344323 | controller |     - '        '
2026-03-14 09:44:27.344328 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.344338 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.344342 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.344346 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.344351 | controller |       --timeout=300s'
2026-03-14 09:44:27.344355 | controller |     - '        '
2026-03-14 09:44:27.344359 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.344363 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.344368 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.344372 | controller |     - '        '
2026-03-14 09:44:27.344388 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.344393 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.344397 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.344402 | controller |     - '        '
2026-03-14 09:44:27.344406 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.344410 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.344414 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.344418 | controller |       available"'
2026-03-14 09:44:27.344423 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.344427 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.344431 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.344435 | controller |     - '          exit 1'
2026-03-14 09:44:27.344440 | controller |     - '        fi'
2026-03-14 09:44:27.344444 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.344448 | controller |     - '        '
2026-03-14 09:44:27.344452 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.344457 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.344461 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.344465 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.344469 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.344473 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.344478 | controller |       -- \'
2026-03-14 09:44:27.344482 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.344486 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.344490 | controller |     - '    logger.go:42: 09:32:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.344495 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.344499 | controller |     - '    logger.go:42: 09:32:35 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.344503 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.344507 | controller |       found'
2026-03-14 09:44:27.344514 | controller |     - '    logger.go:42: 09:32:36 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.344520 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.344524 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.344529 | controller |     - '        '
2026-03-14 09:44:27.344533 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.344537 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.344541 | controller |     - '        '
2026-03-14 09:44:27.344545 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.344550 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.344554 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.344558 | controller |     - '          exit 1'
2026-03-14 09:44:27.344562 | controller |     - '        fi'
2026-03-14 09:44:27.344567 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.344571 | controller |     - '        '
2026-03-14 09:44:27.344577 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.344581 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.344585 | controller |       | base64 -d)'
2026-03-14 09:44:27.344589 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.344593 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.344598 | controller |     - '          exit 1'
2026-03-14 09:44:27.344602 | controller |     - '        fi'
2026-03-14 09:44:27.344606 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.344610 | controller |     - '        '
2026-03-14 09:44:27.344614 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.344626 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.344631 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.344635 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.344639 | controller |       --timeout=300s'
2026-03-14 09:44:27.344643 | controller |     - '        '
2026-03-14 09:44:27.344648 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.344652 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.344656 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.344660 | controller |     - '        '
2026-03-14 09:44:27.344665 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.344669 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.344673 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.344677 | controller |     - '        '
2026-03-14 09:44:27.344681 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.344686 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.344690 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.344694 | controller |       available"'
2026-03-14 09:44:27.344698 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.344703 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.344707 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.344711 | controller |     - '          exit 1'
2026-03-14 09:44:27.344721 | controller |     - '        fi'
2026-03-14 09:44:27.344727 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.344732 | controller |     - '        '
2026-03-14 09:44:27.344736 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.344740 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.344744 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.344749 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.344753 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.344757 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.344761 | controller |       -- \'
2026-03-14 09:44:27.344766 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.344770 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.344774 | controller |     - '    logger.go:42: 09:32:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.344779 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.344783 | controller |     - '    logger.go:42: 09:32:36 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.344787 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.344791 | controller |       found'
2026-03-14 09:44:27.344796 | controller |     - '    logger.go:42: 09:32:37 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.344800 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.344804 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.344809 | controller |     - '        '
2026-03-14 09:44:27.344813 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.344817 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.344822 | controller |     - '        '
2026-03-14 09:44:27.344826 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.344830 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.344834 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.344839 | controller |     - '          exit 1'
2026-03-14 09:44:27.344843 | controller |     - '        fi'
2026-03-14 09:44:27.344847 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.344851 | controller |     - '        '
2026-03-14 09:44:27.344856 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.344860 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.344864 | controller |       | base64 -d)'
2026-03-14 09:44:27.344868 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.344896 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.344900 | controller |     - '          exit 1'
2026-03-14 09:44:27.344904 | controller |     - '        fi'
2026-03-14 09:44:27.344909 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.344913 | controller |     - '        '
2026-03-14 09:44:27.344917 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.344922 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.344935 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.344940 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.344945 | controller |       --timeout=300s'
2026-03-14 09:44:27.344950 | controller |     - '        '
2026-03-14 09:44:27.344957 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.344961 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.344966 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.344970 | controller |     - '        '
2026-03-14 09:44:27.344974 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.344980 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.344984 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.344989 | controller |     - '        '
2026-03-14 09:44:27.344993 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.344997 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.345001 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.345005 | controller |       available"'
2026-03-14 09:44:27.345010 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.345014 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.345018 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.345022 | controller |     - '          exit 1'
2026-03-14 09:44:27.345026 | controller |     - '        fi'
2026-03-14 09:44:27.345031 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.345035 | controller |     - '        '
2026-03-14 09:44:27.345039 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.345043 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.345048 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.345052 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.345056 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.345060 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.345064 | controller |       -- \'
2026-03-14 09:44:27.345069 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.345073 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.345077 | controller |     - '    logger.go:42: 09:32:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.345081 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.345086 | controller |     - '    logger.go:42: 09:32:37 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.345090 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.345094 | controller |       found'
2026-03-14 09:44:27.345098 | controller |     - '    logger.go:42: 09:32:38 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.345102 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.345107 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.345111 | controller |     - '        '
2026-03-14 09:44:27.345117 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.345121 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.345125 | controller |     - '        '
2026-03-14 09:44:27.345129 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.345135 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.345139 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.345143 | controller |     - '          exit 1'
2026-03-14 09:44:27.345148 | controller |     - '        fi'
2026-03-14 09:44:27.345152 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.345156 | controller |     - '        '
2026-03-14 09:44:27.345160 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.345165 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.345169 | controller |       | base64 -d)'
2026-03-14 09:44:27.345173 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.345177 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.345181 | controller |     - '          exit 1'
2026-03-14 09:44:27.345186 | controller |     - '        fi'
2026-03-14 09:44:27.345190 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.345194 | controller |     - '        '
2026-03-14 09:44:27.345199 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.345203 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.345213 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.345218 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.345222 | controller |       --timeout=300s'
2026-03-14 09:44:27.345226 | controller |     - '        '
2026-03-14 09:44:27.345230 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.345235 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.345239 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.345243 | controller |     - '        '
2026-03-14 09:44:27.345248 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.345252 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.345256 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.345260 | controller |     - '        '
2026-03-14 09:44:27.345265 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.345269 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.345273 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.345277 | controller |       available"'
2026-03-14 09:44:27.345281 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.345286 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.345305 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.345310 | controller |     - '          exit 1'
2026-03-14 09:44:27.345314 | controller |     - '        fi'
2026-03-14 09:44:27.345318 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.345323 | controller |     - '        '
2026-03-14 09:44:27.345327 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.345331 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.345335 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.345340 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.345347 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.345352 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.345356 | controller |       -- \'
2026-03-14 09:44:27.345360 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.345364 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.345369 | controller |     - '    logger.go:42: 09:32:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.345373 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.345377 | controller |     - '    logger.go:42: 09:32:38 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.345382 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.345386 | controller |       found'
2026-03-14 09:44:27.345390 | controller |     - '    logger.go:42: 09:32:39 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.345394 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.345399 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.345403 | controller |     - '        '
2026-03-14 09:44:27.345407 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.345411 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.345416 | controller |     - '        '
2026-03-14 09:44:27.345444 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.345449 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.345453 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.345458 | controller |     - '          exit 1'
2026-03-14 09:44:27.345462 | controller |     - '        fi'
2026-03-14 09:44:27.345466 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.345470 | controller |     - '        '
2026-03-14 09:44:27.345474 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.345479 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.345483 | controller |       | base64 -d)'
2026-03-14 09:44:27.345487 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.345494 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.345498 | controller |     - '          exit 1'
2026-03-14 09:44:27.345503 | controller |     - '        fi'
2026-03-14 09:44:27.345507 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.345511 | controller |     - '        '
2026-03-14 09:44:27.345515 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.345520 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.345524 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.345536 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.345540 | controller |       --timeout=300s'
2026-03-14 09:44:27.345545 | controller |     - '        '
2026-03-14 09:44:27.345549 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.345553 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.345557 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.345562 | controller |     - '        '
2026-03-14 09:44:27.345566 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.345572 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.345576 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.345581 | controller |     - '        '
2026-03-14 09:44:27.345585 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.345589 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.345594 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.345598 | controller |       available"'
2026-03-14 09:44:27.345602 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.345606 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.345611 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.345615 | controller |     - '          exit 1'
2026-03-14 09:44:27.345619 | controller |     - '        fi'
2026-03-14 09:44:27.345623 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.345628 | controller |     - '        '
2026-03-14 09:44:27.345632 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.345636 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.345640 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.345645 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.345649 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.345653 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.345657 | controller |       -- \'
2026-03-14 09:44:27.345662 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.345666 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.345670 | controller |     - '    logger.go:42: 09:32:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.345675 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.345679 | controller |     - '    logger.go:42: 09:32:39 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.345683 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.345687 | controller |       found'
2026-03-14 09:44:27.345692 | controller |     - '    logger.go:42: 09:32:40 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.345696 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.345700 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.345704 | controller |     - '        '
2026-03-14 09:44:27.345709 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.345715 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.345720 | controller |     - '        '
2026-03-14 09:44:27.345724 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.345728 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.345732 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.345737 | controller |     - '          exit 1'
2026-03-14 09:44:27.345741 | controller |     - '        fi'
2026-03-14 09:44:27.345745 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.345749 | controller |     - '        '
2026-03-14 09:44:27.345754 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.345758 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.345764 | controller |       | base64 -d)'
2026-03-14 09:44:27.345768 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.345772 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.345777 | controller |     - '          exit 1'
2026-03-14 09:44:27.345781 | controller |     - '        fi'
2026-03-14 09:44:27.345785 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.345789 | controller |     - '        '
2026-03-14 09:44:27.345794 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.345798 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.345802 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.345807 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.345817 | controller |       --timeout=300s'
2026-03-14 09:44:27.345825 | controller |     - '        '
2026-03-14 09:44:27.345829 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.345834 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.345838 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.345842 | controller |     - '        '
2026-03-14 09:44:27.345847 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.345851 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.345855 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.345860 | controller |     - '        '
2026-03-14 09:44:27.345864 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.345868 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.345872 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.345877 | controller |       available"'
2026-03-14 09:44:27.345881 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.345885 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.345890 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.345894 | controller |     - '          exit 1'
2026-03-14 09:44:27.345902 | controller |     - '        fi'
2026-03-14 09:44:27.345907 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.345911 | controller |     - '        '
2026-03-14 09:44:27.345916 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.345921 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.345926 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.345955 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.345960 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.345965 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.345969 | controller |       -- \'
2026-03-14 09:44:27.345974 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.345978 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.345985 | controller |     - '    logger.go:42: 09:32:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.345989 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.345997 | controller |     - '    logger.go:42: 09:32:41 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.346021 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.346026 | controller |       found'
2026-03-14 09:44:27.346030 | controller |     - '    logger.go:42: 09:32:42 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.346034 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.346039 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.346043 | controller |     - '        '
2026-03-14 09:44:27.346048 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.346052 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.346056 | controller |     - '        '
2026-03-14 09:44:27.346060 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.346065 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.346069 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.346073 | controller |     - '          exit 1'
2026-03-14 09:44:27.346078 | controller |     - '        fi'
2026-03-14 09:44:27.346082 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.346087 | controller |     - '        '
2026-03-14 09:44:27.346091 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.346095 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.346100 | controller |       | base64 -d)'
2026-03-14 09:44:27.346104 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.346108 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.346112 | controller |     - '          exit 1'
2026-03-14 09:44:27.346117 | controller |     - '        fi'
2026-03-14 09:44:27.346121 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.346125 | controller |     - '        '
2026-03-14 09:44:27.346130 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.346134 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.346138 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.346143 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.346147 | controller |       --timeout=300s'
2026-03-14 09:44:27.346151 | controller |     - '        '
2026-03-14 09:44:27.346163 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.346168 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.346172 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.346177 | controller |     - '        '
2026-03-14 09:44:27.346181 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.346185 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.346190 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.346194 | controller |     - '        '
2026-03-14 09:44:27.346198 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.346203 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.346207 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.346213 | controller |       available"'
2026-03-14 09:44:27.346217 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.346222 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.346226 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.346230 | controller |     - '          exit 1'
2026-03-14 09:44:27.346235 | controller |     - '        fi'
2026-03-14 09:44:27.346239 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.346243 | controller |     - '        '
2026-03-14 09:44:27.346248 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.346252 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.346256 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.346260 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.346265 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.346269 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.346273 | controller |       -- \'
2026-03-14 09:44:27.346278 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.346282 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.346286 | controller |     - '    logger.go:42: 09:32:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.346305 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.346310 | controller |     - '    logger.go:42: 09:32:42 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.346314 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.346319 | controller |       found'
2026-03-14 09:44:27.346323 | controller |     - '    logger.go:42: 09:32:43 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.346327 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.346332 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.346336 | controller |     - '        '
2026-03-14 09:44:27.346340 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.346345 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.346349 | controller |     - '        '
2026-03-14 09:44:27.346355 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.346360 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.346364 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.346368 | controller |     - '          exit 1'
2026-03-14 09:44:27.346373 | controller |     - '        fi'
2026-03-14 09:44:27.346377 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.346381 | controller |     - '        '
2026-03-14 09:44:27.346386 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.346390 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.346394 | controller |       | base64 -d)'
2026-03-14 09:44:27.346399 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.346403 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.346407 | controller |     - '          exit 1'
2026-03-14 09:44:27.346411 | controller |     - '        fi'
2026-03-14 09:44:27.346416 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.346420 | controller |     - '        '
2026-03-14 09:44:27.346426 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.346430 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.346434 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.346439 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.346443 | controller |       --timeout=300s'
2026-03-14 09:44:27.346447 | controller |     - '        '
2026-03-14 09:44:27.346473 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.346481 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.346488 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.346494 | controller |     - '        '
2026-03-14 09:44:27.346500 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.346507 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.346513 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.346520 | controller |     - '        '
2026-03-14 09:44:27.346527 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.346534 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.346540 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.346546 | controller |       available"'
2026-03-14 09:44:27.346553 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.346558 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.346563 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.346567 | controller |     - '          exit 1'
2026-03-14 09:44:27.346571 | controller |     - '        fi'
2026-03-14 09:44:27.346576 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.346580 | controller |     - '        '
2026-03-14 09:44:27.346584 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.346589 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.346593 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.346597 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.346602 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.346606 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.346610 | controller |       -- \'
2026-03-14 09:44:27.346614 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.346619 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.346623 | controller |     - '    logger.go:42: 09:32:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.346628 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.346634 | controller |     - '    logger.go:42: 09:32:43 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.346640 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.346646 | controller |       found'
2026-03-14 09:44:27.346652 | controller |     - '    logger.go:42: 09:32:44 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.346661 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.346667 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.346673 | controller |     - '        '
2026-03-14 09:44:27.346679 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.346685 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.346691 | controller |     - '        '
2026-03-14 09:44:27.346697 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.346703 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.346709 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.346715 | controller |     - '          exit 1'
2026-03-14 09:44:27.346720 | controller |     - '        fi'
2026-03-14 09:44:27.346726 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.346732 | controller |     - '        '
2026-03-14 09:44:27.346738 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.346744 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.346749 | controller |       | base64 -d)'
2026-03-14 09:44:27.346753 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.346757 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.346761 | controller |     - '          exit 1'
2026-03-14 09:44:27.346766 | controller |     - '        fi'
2026-03-14 09:44:27.346770 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.346774 | controller |     - '        '
2026-03-14 09:44:27.346778 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.346782 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.346787 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.346791 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.346795 | controller |       --timeout=300s'
2026-03-14 09:44:27.346799 | controller |     - '        '
2026-03-14 09:44:27.346803 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.346820 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.346825 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.346829 | controller |     - '        '
2026-03-14 09:44:27.346833 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.346838 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.346842 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.346846 | controller |     - '        '
2026-03-14 09:44:27.346850 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.346854 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.346859 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.346863 | controller |       available"'
2026-03-14 09:44:27.346867 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.346872 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.346876 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.346880 | controller |     - '          exit 1'
2026-03-14 09:44:27.346884 | controller |     - '        fi'
2026-03-14 09:44:27.346889 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.346896 | controller |     - '        '
2026-03-14 09:44:27.346900 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.346905 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.346909 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.346913 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.346917 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.346922 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.346926 | controller |       -- \'
2026-03-14 09:44:27.346930 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.346934 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.346940 | controller |     - '    logger.go:42: 09:32:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.346945 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.346949 | controller |     - '    logger.go:42: 09:32:44 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.346953 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.346958 | controller |       found'
2026-03-14 09:44:27.346962 | controller |     - '    logger.go:42: 09:32:45 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.346966 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.346972 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.346976 | controller |     - '        '
2026-03-14 09:44:27.346981 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.346985 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.346989 | controller |     - '        '
2026-03-14 09:44:27.346994 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.346998 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.347002 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.347006 | controller |     - '          exit 1'
2026-03-14 09:44:27.347011 | controller |     - '        fi'
2026-03-14 09:44:27.347015 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.347021 | controller |     - '        '
2026-03-14 09:44:27.347025 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.347029 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.347034 | controller |       | base64 -d)'
2026-03-14 09:44:27.347038 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.347043 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.347048 | controller |     - '          exit 1'
2026-03-14 09:44:27.347052 | controller |     - '        fi'
2026-03-14 09:44:27.347056 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.347060 | controller |     - '        '
2026-03-14 09:44:27.347064 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.347069 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.347073 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.347077 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.347081 | controller |       --timeout=300s'
2026-03-14 09:44:27.347086 | controller |     - '        '
2026-03-14 09:44:27.347090 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.347101 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.347107 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.347111 | controller |     - '        '
2026-03-14 09:44:27.347116 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.347120 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.347126 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.347130 | controller |     - '        '
2026-03-14 09:44:27.347134 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.347139 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.347143 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.347148 | controller |       available"'
2026-03-14 09:44:27.347153 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.347159 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.347164 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.347170 | controller |     - '          exit 1'
2026-03-14 09:44:27.347176 | controller |     - '        fi'
2026-03-14 09:44:27.347182 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.347187 | controller |     - '        '
2026-03-14 09:44:27.347193 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.347199 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.347204 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.347210 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.347216 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.347221 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.347228 | controller |       -- \'
2026-03-14 09:44:27.347234 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.347240 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.347246 | controller |     - '    logger.go:42: 09:32:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.347252 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.347258 | controller |     - '    logger.go:42: 09:32:45 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.347264 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.347270 | controller |       found'
2026-03-14 09:44:27.347276 | controller |     - '    logger.go:42: 09:32:46 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.347281 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.347303 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.347314 | controller |     - '        '
2026-03-14 09:44:27.347320 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.347326 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.347332 | controller |     - '        '
2026-03-14 09:44:27.347336 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.347343 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.347347 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.347352 | controller |     - '          exit 1'
2026-03-14 09:44:27.347356 | controller |     - '        fi'
2026-03-14 09:44:27.347360 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.347364 | controller |     - '        '
2026-03-14 09:44:27.347372 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.347376 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.347380 | controller |       | base64 -d)'
2026-03-14 09:44:27.347385 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.347389 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.347393 | controller |     - '          exit 1'
2026-03-14 09:44:27.347397 | controller |     - '        fi'
2026-03-14 09:44:27.347401 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.347406 | controller |     - '        '
2026-03-14 09:44:27.347410 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.347414 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.347418 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.347422 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.347427 | controller |       --timeout=300s'
2026-03-14 09:44:27.347431 | controller |     - '        '
2026-03-14 09:44:27.347435 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.347439 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.347452 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.347457 | controller |     - '        '
2026-03-14 09:44:27.347461 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.347466 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.347470 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.347474 | controller |     - '        '
2026-03-14 09:44:27.347478 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.347482 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.347486 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.347491 | controller |       available"'
2026-03-14 09:44:27.347495 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.347499 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.347503 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.347507 | controller |     - '          exit 1'
2026-03-14 09:44:27.347511 | controller |     - '        fi'
2026-03-14 09:44:27.347516 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.347520 | controller |     - '        '
2026-03-14 09:44:27.347524 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.347528 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.347532 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.347536 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.347542 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.347546 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.347551 | controller |       -- \'
2026-03-14 09:44:27.347555 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.347559 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.347563 | controller |     - '    logger.go:42: 09:32:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.347568 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.347572 | controller |     - '    logger.go:42: 09:32:46 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.347576 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.347580 | controller |       found'
2026-03-14 09:44:27.347584 | controller |     - '    logger.go:42: 09:32:47 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.347588 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.347593 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.347597 | controller |     - '        '
2026-03-14 09:44:27.347601 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.347605 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.347610 | controller |     - '        '
2026-03-14 09:44:27.347614 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.347618 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.347622 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.347626 | controller |     - '          exit 1'
2026-03-14 09:44:27.347631 | controller |     - '        fi'
2026-03-14 09:44:27.347636 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.347641 | controller |     - '        '
2026-03-14 09:44:27.347647 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.347653 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.347658 | controller |       | base64 -d)'
2026-03-14 09:44:27.347664 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.347670 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.347676 | controller |     - '          exit 1'
2026-03-14 09:44:27.347682 | controller |     - '        fi'
2026-03-14 09:44:27.347687 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.347693 | controller |     - '        '
2026-03-14 09:44:27.347698 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.347704 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.347712 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.347718 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.347724 | controller |       --timeout=300s'
2026-03-14 09:44:27.347730 | controller |     - '        '
2026-03-14 09:44:27.347735 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.347741 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.347754 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.347761 | controller |     - '        '
2026-03-14 09:44:27.347767 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.347773 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.347781 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.347786 | controller |     - '        '
2026-03-14 09:44:27.347792 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.347798 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.347803 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.347807 | controller |       available"'
2026-03-14 09:44:27.347811 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.347815 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.347819 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.347824 | controller |     - '          exit 1'
2026-03-14 09:44:27.347828 | controller |     - '        fi'
2026-03-14 09:44:27.347832 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.347836 | controller |     - '        '
2026-03-14 09:44:27.347840 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.347845 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.347849 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.347853 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.347857 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.347862 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.347866 | controller |       -- \'
2026-03-14 09:44:27.347870 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.347874 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.347878 | controller |     - '    logger.go:42: 09:32:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.347883 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.347887 | controller |     - '    logger.go:42: 09:32:47 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.347891 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.347895 | controller |       found'
2026-03-14 09:44:27.347899 | controller |     - '    logger.go:42: 09:32:48 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.347904 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.347908 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.347912 | controller |     - '        '
2026-03-14 09:44:27.347916 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.347921 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.347925 | controller |     - '        '
2026-03-14 09:44:27.347929 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.347933 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.347937 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.347942 | controller |     - '          exit 1'
2026-03-14 09:44:27.347946 | controller |     - '        fi'
2026-03-14 09:44:27.347950 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.347954 | controller |     - '        '
2026-03-14 09:44:27.347958 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.347963 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.347968 | controller |       | base64 -d)'
2026-03-14 09:44:27.347973 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.347977 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.347981 | controller |     - '          exit 1'
2026-03-14 09:44:27.347985 | controller |     - '        fi'
2026-03-14 09:44:27.347989 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.347994 | controller |     - '        '
2026-03-14 09:44:27.347998 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.348002 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.348006 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.348010 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.348015 | controller |       --timeout=300s'
2026-03-14 09:44:27.348019 | controller |     - '        '
2026-03-14 09:44:27.348023 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.348027 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.348038 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.348042 | controller |     - '        '
2026-03-14 09:44:27.348047 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.348051 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.348055 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.348059 | controller |     - '        '
2026-03-14 09:44:27.348064 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.348068 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.348072 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.348076 | controller |       available"'
2026-03-14 09:44:27.348081 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.348085 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.348089 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.348093 | controller |     - '          exit 1'
2026-03-14 09:44:27.348097 | controller |     - '        fi'
2026-03-14 09:44:27.348102 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.348106 | controller |     - '        '
2026-03-14 09:44:27.348113 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.348117 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.348121 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.348125 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.348130 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.348134 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.348138 | controller |       -- \'
2026-03-14 09:44:27.348142 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.348147 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.348152 | controller |     - '    logger.go:42: 09:32:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.348160 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.348164 | controller |     - '    logger.go:42: 09:32:48 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.348168 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.348173 | controller |       found'
2026-03-14 09:44:27.348177 | controller |     - '    logger.go:42: 09:32:50 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.348181 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.348187 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.348193 | controller |     - '        '
2026-03-14 09:44:27.348199 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.348205 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.348210 | controller |     - '        '
2026-03-14 09:44:27.348216 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.348222 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.348228 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.348234 | controller |     - '          exit 1'
2026-03-14 09:44:27.348238 | controller |     - '        fi'
2026-03-14 09:44:27.348243 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.348247 | controller |     - '        '
2026-03-14 09:44:27.348251 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.348255 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.348260 | controller |       | base64 -d)'
2026-03-14 09:44:27.348264 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.348268 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.348273 | controller |     - '          exit 1'
2026-03-14 09:44:27.348277 | controller |     - '        fi'
2026-03-14 09:44:27.348281 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.348285 | controller |     - '        '
2026-03-14 09:44:27.348306 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.348310 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.348315 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.348319 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.348326 | controller |       --timeout=300s'
2026-03-14 09:44:27.348330 | controller |     - '        '
2026-03-14 09:44:27.348334 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.348339 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.348343 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.348347 | controller |     - '        '
2026-03-14 09:44:27.348360 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.348365 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.348369 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.348373 | controller |     - '        '
2026-03-14 09:44:27.348378 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.348382 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.348386 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.348392 | controller |       available"'
2026-03-14 09:44:27.348397 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.348401 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.348405 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.348409 | controller |     - '          exit 1'
2026-03-14 09:44:27.348414 | controller |     - '        fi'
2026-03-14 09:44:27.348418 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.348422 | controller |     - '        '
2026-03-14 09:44:27.348426 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.348431 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.348435 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.348439 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.348443 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.348447 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.348452 | controller |       -- \'
2026-03-14 09:44:27.348456 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.348460 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.348464 | controller |     - '    logger.go:42: 09:32:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.348469 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.348473 | controller |     - '    logger.go:42: 09:32:50 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.348477 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.348481 | controller |       found'
2026-03-14 09:44:27.348485 | controller |     - '    logger.go:42: 09:32:51 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.348490 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.348494 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.348498 | controller |     - '        '
2026-03-14 09:44:27.348502 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.348507 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.348511 | controller |     - '        '
2026-03-14 09:44:27.348515 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.348519 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.348523 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.348528 | controller |     - '          exit 1'
2026-03-14 09:44:27.348532 | controller |     - '        fi'
2026-03-14 09:44:27.348536 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.348540 | controller |     - '        '
2026-03-14 09:44:27.348545 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.348549 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.348553 | controller |       | base64 -d)'
2026-03-14 09:44:27.348557 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.348562 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.348566 | controller |     - '          exit 1'
2026-03-14 09:44:27.348570 | controller |     - '        fi'
2026-03-14 09:44:27.348574 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.348579 | controller |     - '        '
2026-03-14 09:44:27.348583 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.348589 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.348593 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.348597 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.348601 | controller |       --timeout=300s'
2026-03-14 09:44:27.348605 | controller |     - '        '
2026-03-14 09:44:27.348610 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.348614 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.348618 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.348622 | controller |     - '        '
2026-03-14 09:44:27.348632 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.348637 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.348641 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.348645 | controller |     - '        '
2026-03-14 09:44:27.348649 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.348654 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.348658 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.348662 | controller |       available"'
2026-03-14 09:44:27.348666 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.348672 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.348677 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.348681 | controller |     - '          exit 1'
2026-03-14 09:44:27.348685 | controller |     - '        fi'
2026-03-14 09:44:27.348689 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.348694 | controller |     - '        '
2026-03-14 09:44:27.348698 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.348702 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.348706 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.348710 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.348715 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.348719 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.348723 | controller |       -- \'
2026-03-14 09:44:27.348727 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.348732 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.348736 | controller |     - '    logger.go:42: 09:32:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.348740 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.348744 | controller |     - '    logger.go:42: 09:32:51 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.348749 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.348753 | controller |       found'
2026-03-14 09:44:27.348757 | controller |     - '    logger.go:42: 09:32:52 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.348761 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.348767 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.348772 | controller |     - '        '
2026-03-14 09:44:27.348776 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.348780 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.348784 | controller |     - '        '
2026-03-14 09:44:27.348788 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.348793 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.348797 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.348801 | controller |     - '          exit 1'
2026-03-14 09:44:27.348805 | controller |     - '        fi'
2026-03-14 09:44:27.348810 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.348814 | controller |     - '        '
2026-03-14 09:44:27.348818 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.348822 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.348826 | controller |       | base64 -d)'
2026-03-14 09:44:27.348831 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.348835 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.348839 | controller |     - '          exit 1'
2026-03-14 09:44:27.348843 | controller |     - '        fi'
2026-03-14 09:44:27.348848 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.348852 | controller |     - '        '
2026-03-14 09:44:27.348856 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.348860 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.348864 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.348869 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.348873 | controller |       --timeout=300s'
2026-03-14 09:44:27.348877 | controller |     - '        '
2026-03-14 09:44:27.348881 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.348886 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.348890 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.348894 | controller |     - '        '
2026-03-14 09:44:27.348898 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.348910 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.348915 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.348919 | controller |     - '        '
2026-03-14 09:44:27.348924 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.348928 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.348932 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.348936 | controller |       available"'
2026-03-14 09:44:27.348940 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.348945 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.348949 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.348953 | controller |     - '          exit 1'
2026-03-14 09:44:27.348958 | controller |     - '        fi'
2026-03-14 09:44:27.348963 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.348968 | controller |     - '        '
2026-03-14 09:44:27.348974 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.348980 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.348985 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.348989 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.348994 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.348998 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.349002 | controller |       -- \'
2026-03-14 09:44:27.349006 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.349011 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.349015 | controller |     - '    logger.go:42: 09:32:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.349019 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.349023 | controller |     - '    logger.go:42: 09:32:52 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.349028 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.349032 | controller |       found'
2026-03-14 09:44:27.349036 | controller |     - '    logger.go:42: 09:32:53 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.349040 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.349044 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.349049 | controller |     - '        '
2026-03-14 09:44:27.349053 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.349057 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.349061 | controller |     - '        '
2026-03-14 09:44:27.349066 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.349070 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.349074 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.349078 | controller |     - '          exit 1'
2026-03-14 09:44:27.349083 | controller |     - '        fi'
2026-03-14 09:44:27.349087 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.349091 | controller |     - '        '
2026-03-14 09:44:27.349095 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.349100 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.349104 | controller |       | base64 -d)'
2026-03-14 09:44:27.349108 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.349112 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.349117 | controller |     - '          exit 1'
2026-03-14 09:44:27.349121 | controller |     - '        fi'
2026-03-14 09:44:27.349125 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.349129 | controller |     - '        '
2026-03-14 09:44:27.349133 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.349138 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.349142 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.349146 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.349150 | controller |       --timeout=300s'
2026-03-14 09:44:27.349155 | controller |     - '        '
2026-03-14 09:44:27.349159 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.349165 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.349169 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.349173 | controller |     - '        '
2026-03-14 09:44:27.349179 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.349188 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.349193 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.349197 | controller |     - '        '
2026-03-14 09:44:27.349201 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.349207 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.349211 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.349216 | controller |       available"'
2026-03-14 09:44:27.349220 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.349224 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.349228 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.349233 | controller |     - '          exit 1'
2026-03-14 09:44:27.349237 | controller |     - '        fi'
2026-03-14 09:44:27.349243 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.349248 | controller |     - '        '
2026-03-14 09:44:27.349252 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.349256 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.349260 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.349265 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.349269 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.349273 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.349277 | controller |       -- \'
2026-03-14 09:44:27.349282 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.349286 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.349387 | controller |     - '    logger.go:42: 09:32:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.349393 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.349397 | controller |     - '    logger.go:42: 09:32:53 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.349402 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.349406 | controller |       found'
2026-03-14 09:44:27.349410 | controller |     - '    logger.go:42: 09:32:54 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.349414 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.349419 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.349423 | controller |     - '        '
2026-03-14 09:44:27.349427 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.349432 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.349436 | controller |     - '        '
2026-03-14 09:44:27.349442 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.349448 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.349457 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.349464 | controller |     - '          exit 1'
2026-03-14 09:44:27.349470 | controller |     - '        fi'
2026-03-14 09:44:27.349475 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.349481 | controller |     - '        '
2026-03-14 09:44:27.349487 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.349493 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.349499 | controller |       | base64 -d)'
2026-03-14 09:44:27.349505 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.349510 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.349515 | controller |     - '          exit 1'
2026-03-14 09:44:27.349519 | controller |     - '        fi'
2026-03-14 09:44:27.349523 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.349527 | controller |     - '        '
2026-03-14 09:44:27.349532 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.349536 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.349540 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.349545 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.349549 | controller |       --timeout=300s'
2026-03-14 09:44:27.349553 | controller |     - '        '
2026-03-14 09:44:27.349557 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.349562 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.349566 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.349570 | controller |     - '        '
2026-03-14 09:44:27.349574 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.349579 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.349593 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.349598 | controller |     - '        '
2026-03-14 09:44:27.349602 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.349607 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.349615 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.349619 | controller |       available"'
2026-03-14 09:44:27.349623 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.349628 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.349632 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.349638 | controller |     - '          exit 1'
2026-03-14 09:44:27.349643 | controller |     - '        fi'
2026-03-14 09:44:27.349649 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.349655 | controller |     - '        '
2026-03-14 09:44:27.349661 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.349666 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.349672 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.349678 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.349684 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.349693 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.349699 | controller |       -- \'
2026-03-14 09:44:27.349705 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.349710 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.349716 | controller |     - '    logger.go:42: 09:32:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.349721 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.349726 | controller |     - '    logger.go:42: 09:32:54 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.349731 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.349735 | controller |       found'
2026-03-14 09:44:27.349739 | controller |     - '    logger.go:42: 09:32:55 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.349744 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.349748 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.349752 | controller |     - '        '
2026-03-14 09:44:27.349756 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.349761 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.349765 | controller |     - '        '
2026-03-14 09:44:27.349769 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.349774 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.349778 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.349782 | controller |     - '          exit 1'
2026-03-14 09:44:27.349787 | controller |     - '        fi'
2026-03-14 09:44:27.349791 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.349795 | controller |     - '        '
2026-03-14 09:44:27.349799 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.349804 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.349808 | controller |       | base64 -d)'
2026-03-14 09:44:27.349812 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.349818 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.349823 | controller |     - '          exit 1'
2026-03-14 09:44:27.349827 | controller |     - '        fi'
2026-03-14 09:44:27.349831 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.349836 | controller |     - '        '
2026-03-14 09:44:27.349840 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.349844 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.349849 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.349853 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.349857 | controller |       --timeout=300s'
2026-03-14 09:44:27.349861 | controller |     - '        '
2026-03-14 09:44:27.349866 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.349870 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.349874 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.349878 | controller |     - '        '
2026-03-14 09:44:27.349883 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.349887 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.349900 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.349905 | controller |     - '        '
2026-03-14 09:44:27.349910 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.349915 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.349919 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.349923 | controller |       available"'
2026-03-14 09:44:27.349928 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.349932 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.349936 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.349940 | controller |     - '          exit 1'
2026-03-14 09:44:27.349945 | controller |     - '        fi'
2026-03-14 09:44:27.349949 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.349953 | controller |     - '        '
2026-03-14 09:44:27.349958 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.349962 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.349968 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.349973 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.349977 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.349982 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.349988 | controller |       -- \'
2026-03-14 09:44:27.349993 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.349997 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.350001 | controller |     - '    logger.go:42: 09:32:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.350005 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.350010 | controller |     - '    logger.go:42: 09:32:56 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.350029 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.350033 | controller |       found'
2026-03-14 09:44:27.350037 | controller |     - '    logger.go:42: 09:32:57 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.350042 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.350046 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.350050 | controller |     - '        '
2026-03-14 09:44:27.350055 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.350059 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.350063 | controller |     - '        '
2026-03-14 09:44:27.350068 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.350072 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.350076 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.350080 | controller |     - '          exit 1'
2026-03-14 09:44:27.350085 | controller |     - '        fi'
2026-03-14 09:44:27.350089 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.350093 | controller |     - '        '
2026-03-14 09:44:27.350101 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.350105 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.350109 | controller |       | base64 -d)'
2026-03-14 09:44:27.350115 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.350120 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.350124 | controller |     - '          exit 1'
2026-03-14 09:44:27.350128 | controller |     - '        fi'
2026-03-14 09:44:27.350132 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.350137 | controller |     - '        '
2026-03-14 09:44:27.350141 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.350145 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.350150 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.350154 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.350158 | controller |       --timeout=300s'
2026-03-14 09:44:27.350164 | controller |     - '        '
2026-03-14 09:44:27.350168 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.350173 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.350177 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.350181 | controller |     - '        '
2026-03-14 09:44:27.350185 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.350190 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.350194 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.350204 | controller |     - '        '
2026-03-14 09:44:27.350209 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.350213 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.350218 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.350222 | controller |       available"'
2026-03-14 09:44:27.350226 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.350231 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.350235 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.350239 | controller |     - '          exit 1'
2026-03-14 09:44:27.350243 | controller |     - '        fi'
2026-03-14 09:44:27.350248 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.350252 | controller |     - '        '
2026-03-14 09:44:27.350256 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.350261 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.350265 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.350269 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.350273 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.350278 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.350282 | controller |       -- \'
2026-03-14 09:44:27.350286 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.350307 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.350311 | controller |     - '    logger.go:42: 09:32:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.350317 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.350324 | controller |     - '    logger.go:42: 09:32:57 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.350328 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.350332 | controller |       found'
2026-03-14 09:44:27.350336 | controller |     - '    logger.go:42: 09:32:58 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.350341 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.350345 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.350349 | controller |     - '        '
2026-03-14 09:44:27.350353 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.350358 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.350362 | controller |     - '        '
2026-03-14 09:44:27.350366 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.350370 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.350374 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.350379 | controller |     - '          exit 1'
2026-03-14 09:44:27.350383 | controller |     - '        fi'
2026-03-14 09:44:27.350387 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.350391 | controller |     - '        '
2026-03-14 09:44:27.350396 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.350400 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.350404 | controller |       | base64 -d)'
2026-03-14 09:44:27.350408 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.350413 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.350417 | controller |     - '          exit 1'
2026-03-14 09:44:27.350422 | controller |     - '        fi'
2026-03-14 09:44:27.350427 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.350431 | controller |     - '        '
2026-03-14 09:44:27.350435 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.350439 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.350444 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.350448 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.350452 | controller |       --timeout=300s'
2026-03-14 09:44:27.350456 | controller |     - '        '
2026-03-14 09:44:27.350460 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.350465 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.350470 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.350474 | controller |     - '        '
2026-03-14 09:44:27.350479 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.350483 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.350487 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.350491 | controller |     - '        '
2026-03-14 09:44:27.350503 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.350507 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.350512 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.350516 | controller |       available"'
2026-03-14 09:44:27.350522 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.350526 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.350530 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.350534 | controller |     - '          exit 1'
2026-03-14 09:44:27.350539 | controller |     - '        fi'
2026-03-14 09:44:27.350543 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.350547 | controller |     - '        '
2026-03-14 09:44:27.350551 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.350555 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.350560 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.350564 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.350568 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.350572 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.350576 | controller |       -- \'
2026-03-14 09:44:27.350580 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.350585 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.350589 | controller |     - '    logger.go:42: 09:32:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.350593 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.350597 | controller |     - '    logger.go:42: 09:32:58 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.350601 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.350606 | controller |       found'
2026-03-14 09:44:27.350610 | controller |     - '    logger.go:42: 09:32:59 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.350614 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.350618 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.350623 | controller |     - '        '
2026-03-14 09:44:27.350627 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.350631 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.350636 | controller |     - '        '
2026-03-14 09:44:27.350641 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.350647 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.350655 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.350661 | controller |     - '          exit 1'
2026-03-14 09:44:27.350667 | controller |     - '        fi'
2026-03-14 09:44:27.350673 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.350679 | controller |     - '        '
2026-03-14 09:44:27.350684 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.350690 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.350695 | controller |       | base64 -d)'
2026-03-14 09:44:27.350701 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.350707 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.350711 | controller |     - '          exit 1'
2026-03-14 09:44:27.350715 | controller |     - '        fi'
2026-03-14 09:44:27.350719 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.350724 | controller |     - '        '
2026-03-14 09:44:27.350728 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.350732 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.350738 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.350743 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.350747 | controller |       --timeout=300s'
2026-03-14 09:44:27.350751 | controller |     - '        '
2026-03-14 09:44:27.350755 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.350760 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.350764 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.350768 | controller |     - '        '
2026-03-14 09:44:27.350772 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.350776 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.350781 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.350785 | controller |     - '        '
2026-03-14 09:44:27.350789 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.350801 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.350806 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.350811 | controller |       available"'
2026-03-14 09:44:27.350816 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.350820 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.350824 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.350829 | controller |     - '          exit 1'
2026-03-14 09:44:27.350833 | controller |     - '        fi'
2026-03-14 09:44:27.350837 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.350842 | controller |     - '        '
2026-03-14 09:44:27.350846 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.350852 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.350856 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.350861 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.350865 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.350869 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.350874 | controller |       -- \'
2026-03-14 09:44:27.350879 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.350883 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.350887 | controller |     - '    logger.go:42: 09:32:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.350892 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.350896 | controller |     - '    logger.go:42: 09:32:59 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.350900 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.350905 | controller |       found'
2026-03-14 09:44:27.350909 | controller |     - '    logger.go:42: 09:33:00 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.350913 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.350917 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.350923 | controller |     - '        '
2026-03-14 09:44:27.350928 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.350932 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.350936 | controller |     - '        '
2026-03-14 09:44:27.350940 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.350945 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.350949 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.350953 | controller |     - '          exit 1'
2026-03-14 09:44:27.350957 | controller |     - '        fi'
2026-03-14 09:44:27.350962 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.350966 | controller |     - '        '
2026-03-14 09:44:27.350970 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.350974 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.350978 | controller |       | base64 -d)'
2026-03-14 09:44:27.350983 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.350987 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.350991 | controller |     - '          exit 1'
2026-03-14 09:44:27.350995 | controller |     - '        fi'
2026-03-14 09:44:27.351000 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.351004 | controller |     - '        '
2026-03-14 09:44:27.351008 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.351012 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.351017 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.351021 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.351025 | controller |       --timeout=300s'
2026-03-14 09:44:27.351029 | controller |     - '        '
2026-03-14 09:44:27.351033 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.351038 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.351042 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.351046 | controller |     - '        '
2026-03-14 09:44:27.351050 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.351055 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.351059 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.351063 | controller |     - '        '
2026-03-14 09:44:27.351067 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.351077 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.351082 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.351086 | controller |       available"'
2026-03-14 09:44:27.351090 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.351094 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.351099 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.351103 | controller |     - '          exit 1'
2026-03-14 09:44:27.351107 | controller |     - '        fi'
2026-03-14 09:44:27.351111 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.351115 | controller |     - '        '
2026-03-14 09:44:27.351120 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.351127 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.351131 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.351135 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.351139 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.351144 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.351148 | controller |       -- \'
2026-03-14 09:44:27.351152 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.351156 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.351162 | controller |     - '    logger.go:42: 09:33:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.351166 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.351171 | controller |     - '    logger.go:42: 09:33:00 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.351175 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.351179 | controller |       found'
2026-03-14 09:44:27.351183 | controller |     - '    logger.go:42: 09:33:01 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.351187 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.351192 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.351196 | controller |     - '        '
2026-03-14 09:44:27.351200 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.351205 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.351210 | controller |     - '        '
2026-03-14 09:44:27.351214 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.351218 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.351222 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.351227 | controller |     - '          exit 1'
2026-03-14 09:44:27.351231 | controller |     - '        fi'
2026-03-14 09:44:27.351235 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.351239 | controller |     - '        '
2026-03-14 09:44:27.351243 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.351248 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.351252 | controller |       | base64 -d)'
2026-03-14 09:44:27.351256 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.351260 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.351264 | controller |     - '          exit 1'
2026-03-14 09:44:27.351269 | controller |     - '        fi'
2026-03-14 09:44:27.351273 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.351277 | controller |     - '        '
2026-03-14 09:44:27.351281 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.351285 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.351304 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.351311 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.351315 | controller |       --timeout=300s'
2026-03-14 09:44:27.351319 | controller |     - '        '
2026-03-14 09:44:27.351324 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.351328 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.351334 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.351338 | controller |     - '        '
2026-03-14 09:44:27.351342 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.351346 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.351351 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.351355 | controller |     - '        '
2026-03-14 09:44:27.351359 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.351363 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.351374 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.351379 | controller |       available"'
2026-03-14 09:44:27.351384 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.351390 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.351394 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.351399 | controller |     - '          exit 1'
2026-03-14 09:44:27.351403 | controller |     - '        fi'
2026-03-14 09:44:27.351407 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.351411 | controller |     - '        '
2026-03-14 09:44:27.351415 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.351420 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.351424 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.351430 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.351435 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.351439 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.351443 | controller |       -- \'
2026-03-14 09:44:27.351448 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.351452 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.351456 | controller |     - '    logger.go:42: 09:33:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.351460 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.351464 | controller |     - '    logger.go:42: 09:33:01 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.351469 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.351473 | controller |       found'
2026-03-14 09:44:27.351477 | controller |     - '    logger.go:42: 09:33:02 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.351481 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.351485 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.351490 | controller |     - '        '
2026-03-14 09:44:27.351494 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.351498 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.351502 | controller |     - '        '
2026-03-14 09:44:27.351508 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.351512 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.351516 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.351522 | controller |     - '          exit 1'
2026-03-14 09:44:27.351527 | controller |     - '        fi'
2026-03-14 09:44:27.351531 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.351535 | controller |     - '        '
2026-03-14 09:44:27.351539 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.351544 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.351548 | controller |       | base64 -d)'
2026-03-14 09:44:27.351552 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.351556 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.351560 | controller |     - '          exit 1'
2026-03-14 09:44:27.351565 | controller |     - '        fi'
2026-03-14 09:44:27.351569 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.351573 | controller |     - '        '
2026-03-14 09:44:27.351577 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.351581 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.351586 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.351590 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.351594 | controller |       --timeout=300s'
2026-03-14 09:44:27.351598 | controller |     - '        '
2026-03-14 09:44:27.351602 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.351607 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.351611 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.351615 | controller |     - '        '
2026-03-14 09:44:27.351619 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.351623 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.351628 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.351632 | controller |     - '        '
2026-03-14 09:44:27.351638 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.351644 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.351650 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.351662 | controller |       available"'
2026-03-14 09:44:27.351668 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.351674 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.351679 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.351685 | controller |     - '          exit 1'
2026-03-14 09:44:27.351690 | controller |     - '        fi'
2026-03-14 09:44:27.351696 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.351701 | controller |     - '        '
2026-03-14 09:44:27.351707 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.351712 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.351718 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.351723 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.351729 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.351736 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.351742 | controller |       -- \'
2026-03-14 09:44:27.351748 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.351754 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.351759 | controller |     - '    logger.go:42: 09:33:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.351765 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.351770 | controller |     - '    logger.go:42: 09:33:02 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.351776 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.351781 | controller |       found'
2026-03-14 09:44:27.351789 | controller |     - '    logger.go:42: 09:33:03 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.351795 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.351801 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.351806 | controller |     - '        '
2026-03-14 09:44:27.351812 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.351818 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.351824 | controller |     - '        '
2026-03-14 09:44:27.351830 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.351836 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.351841 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.351847 | controller |     - '          exit 1'
2026-03-14 09:44:27.351852 | controller |     - '        fi'
2026-03-14 09:44:27.351858 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.351864 | controller |     - '        '
2026-03-14 09:44:27.351870 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.351876 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.351881 | controller |       | base64 -d)'
2026-03-14 09:44:27.351887 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.351892 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.351898 | controller |     - '          exit 1'
2026-03-14 09:44:27.351903 | controller |     - '        fi'
2026-03-14 09:44:27.351909 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.351914 | controller |     - '        '
2026-03-14 09:44:27.351920 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.351926 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.351931 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.351935 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.351940 | controller |       --timeout=300s'
2026-03-14 09:44:27.351945 | controller |     - '        '
2026-03-14 09:44:27.351951 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.351955 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.351959 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.351963 | controller |     - '        '
2026-03-14 09:44:27.351967 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.351971 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.351976 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.351987 | controller |     - '        '
2026-03-14 09:44:27.351991 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.351995 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.352000 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.352004 | controller |       available"'
2026-03-14 09:44:27.352016 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.352020 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.352025 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.352029 | controller |     - '          exit 1'
2026-03-14 09:44:27.352033 | controller |     - '        fi'
2026-03-14 09:44:27.352037 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.352042 | controller |     - '        '
2026-03-14 09:44:27.352046 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.352050 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.352054 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.352058 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.352063 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.352067 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.352071 | controller |       -- \'
2026-03-14 09:44:27.352075 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.352083 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.352087 | controller |     - '    logger.go:42: 09:33:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.352093 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.352098 | controller |     - '    logger.go:42: 09:33:04 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.352104 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.352110 | controller |       found'
2026-03-14 09:44:27.352116 | controller |     - '    logger.go:42: 09:33:05 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.352122 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.352128 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.352133 | controller |     - '        '
2026-03-14 09:44:27.352139 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.352144 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.352149 | controller |     - '        '
2026-03-14 09:44:27.352155 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.352160 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.352166 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.352172 | controller |     - '          exit 1'
2026-03-14 09:44:27.352176 | controller |     - '        fi'
2026-03-14 09:44:27.352180 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.352185 | controller |     - '        '
2026-03-14 09:44:27.352189 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.352193 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.352197 | controller |       | base64 -d)'
2026-03-14 09:44:27.352202 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.352209 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.352213 | controller |     - '          exit 1'
2026-03-14 09:44:27.352217 | controller |     - '        fi'
2026-03-14 09:44:27.352221 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.352226 | controller |     - '        '
2026-03-14 09:44:27.352230 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.352234 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.352238 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.352242 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.352247 | controller |       --timeout=300s'
2026-03-14 09:44:27.352251 | controller |     - '        '
2026-03-14 09:44:27.352255 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.352259 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.352264 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.352268 | controller |     - '        '
2026-03-14 09:44:27.352272 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.352276 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.352281 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.352285 | controller |     - '        '
2026-03-14 09:44:27.352305 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.352310 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.352314 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.352318 | controller |       available"'
2026-03-14 09:44:27.352323 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.352334 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.352339 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.352343 | controller |     - '          exit 1'
2026-03-14 09:44:27.352347 | controller |     - '        fi'
2026-03-14 09:44:27.352351 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.352356 | controller |     - '        '
2026-03-14 09:44:27.352360 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.352364 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.352368 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.352373 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.352377 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.352381 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.352385 | controller |       -- \'
2026-03-14 09:44:27.352390 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.352394 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.352398 | controller |     - '    logger.go:42: 09:33:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.352402 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.352409 | controller |     - '    logger.go:42: 09:33:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.352413 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.352417 | controller |       found'
2026-03-14 09:44:27.352421 | controller |     - '    logger.go:42: 09:33:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.352426 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.352430 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.352434 | controller |     - '        '
2026-03-14 09:44:27.352438 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.352442 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.352447 | controller |     - '        '
2026-03-14 09:44:27.352451 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.352455 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.352459 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.352464 | controller |     - '          exit 1'
2026-03-14 09:44:27.352468 | controller |     - '        fi'
2026-03-14 09:44:27.352472 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.352476 | controller |     - '        '
2026-03-14 09:44:27.352481 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.352485 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.352489 | controller |       | base64 -d)'
2026-03-14 09:44:27.352493 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.352498 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.352502 | controller |     - '          exit 1'
2026-03-14 09:44:27.352506 | controller |     - '        fi'
2026-03-14 09:44:27.352510 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.352515 | controller |     - '        '
2026-03-14 09:44:27.352519 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.352523 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.352527 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.352532 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.352536 | controller |       --timeout=300s'
2026-03-14 09:44:27.352540 | controller |     - '        '
2026-03-14 09:44:27.352544 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.352549 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.352553 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.352557 | controller |     - '        '
2026-03-14 09:44:27.352564 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.352571 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.352575 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.352580 | controller |     - '        '
2026-03-14 09:44:27.352584 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.352588 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.352592 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.352597 | controller |       available"'
2026-03-14 09:44:27.352602 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.352609 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.352620 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.352625 | controller |     - '          exit 1'
2026-03-14 09:44:27.352629 | controller |     - '        fi'
2026-03-14 09:44:27.352634 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.352639 | controller |     - '        '
2026-03-14 09:44:27.352645 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.352651 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.352657 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.352663 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.352668 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.352674 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.352679 | controller |       -- \'
2026-03-14 09:44:27.352685 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.352690 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.352696 | controller |     - '    logger.go:42: 09:33:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.352702 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.352708 | controller |     - '    logger.go:42: 09:33:06 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.352713 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.352717 | controller |       found'
2026-03-14 09:44:27.352721 | controller |     - '    logger.go:42: 09:33:07 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.352726 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.352730 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.352734 | controller |     - '        '
2026-03-14 09:44:27.352738 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.352743 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.352747 | controller |     - '        '
2026-03-14 09:44:27.352751 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.352755 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.352760 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.352764 | controller |     - '          exit 1'
2026-03-14 09:44:27.352768 | controller |     - '        fi'
2026-03-14 09:44:27.352772 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.352777 | controller |     - '        '
2026-03-14 09:44:27.352782 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.352787 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.352791 | controller |       | base64 -d)'
2026-03-14 09:44:27.352796 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.352800 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.352805 | controller |     - '          exit 1'
2026-03-14 09:44:27.352809 | controller |     - '        fi'
2026-03-14 09:44:27.352813 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.352817 | controller |     - '        '
2026-03-14 09:44:27.352822 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.352826 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.352832 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.352837 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.352841 | controller |       --timeout=300s'
2026-03-14 09:44:27.352845 | controller |     - '        '
2026-03-14 09:44:27.352849 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.352854 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.352858 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.352862 | controller |     - '        '
2026-03-14 09:44:27.352866 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.352870 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.352875 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.352879 | controller |     - '        '
2026-03-14 09:44:27.352883 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.352887 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.352892 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.352896 | controller |       available"'
2026-03-14 09:44:27.352900 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.352904 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.352916 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.352920 | controller |     - '          exit 1'
2026-03-14 09:44:27.352925 | controller |     - '        fi'
2026-03-14 09:44:27.352929 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.352933 | controller |     - '        '
2026-03-14 09:44:27.352937 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.352942 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.352946 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.352950 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.352954 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.352958 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.352963 | controller |       -- \'
2026-03-14 09:44:27.352967 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.352971 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.352975 | controller |     - '    logger.go:42: 09:33:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.352979 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.352984 | controller |     - '    logger.go:42: 09:33:07 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.352988 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.352992 | controller |       found'
2026-03-14 09:44:27.352996 | controller |     - '    logger.go:42: 09:33:08 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.353001 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.353005 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.353009 | controller |     - '        '
2026-03-14 09:44:27.353013 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.353020 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.353024 | controller |     - '        '
2026-03-14 09:44:27.353028 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.353032 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.353036 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.353041 | controller |     - '          exit 1'
2026-03-14 09:44:27.353045 | controller |     - '        fi'
2026-03-14 09:44:27.353049 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.353053 | controller |     - '        '
2026-03-14 09:44:27.353057 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.353062 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.353066 | controller |       | base64 -d)'
2026-03-14 09:44:27.353070 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.353074 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.353078 | controller |     - '          exit 1'
2026-03-14 09:44:27.353083 | controller |     - '        fi'
2026-03-14 09:44:27.353087 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.353091 | controller |     - '        '
2026-03-14 09:44:27.353095 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.353099 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.353104 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.353108 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.353112 | controller |       --timeout=300s'
2026-03-14 09:44:27.353116 | controller |     - '        '
2026-03-14 09:44:27.353120 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.353125 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.353129 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.353133 | controller |     - '        '
2026-03-14 09:44:27.353137 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.353141 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.353146 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.353150 | controller |     - '        '
2026-03-14 09:44:27.353154 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.353158 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.353162 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.353167 | controller |       available"'
2026-03-14 09:44:27.353171 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.353175 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.353179 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.353183 | controller |     - '          exit 1'
2026-03-14 09:44:27.353192 | controller |     - '        fi'
2026-03-14 09:44:27.353197 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.353201 | controller |     - '        '
2026-03-14 09:44:27.353205 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.353211 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.353216 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.353220 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.353224 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.353228 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.353232 | controller |       -- \'
2026-03-14 09:44:27.353237 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.353241 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.353245 | controller |     - '    logger.go:42: 09:33:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.353249 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.353253 | controller |     - '    logger.go:42: 09:33:08 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.353260 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.353265 | controller |       found'
2026-03-14 09:44:27.353269 | controller |     - '    logger.go:42: 09:33:09 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.353273 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.353277 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.353281 | controller |     - '        '
2026-03-14 09:44:27.353286 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.353304 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.353309 | controller |     - '        '
2026-03-14 09:44:27.353313 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.353317 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.353322 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.353326 | controller |     - '          exit 1'
2026-03-14 09:44:27.353330 | controller |     - '        fi'
2026-03-14 09:44:27.353334 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.353338 | controller |     - '        '
2026-03-14 09:44:27.353343 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.353347 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.353351 | controller |       | base64 -d)'
2026-03-14 09:44:27.353355 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.353359 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.353364 | controller |     - '          exit 1'
2026-03-14 09:44:27.353368 | controller |     - '        fi'
2026-03-14 09:44:27.353372 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.353376 | controller |     - '        '
2026-03-14 09:44:27.353380 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.353385 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.353389 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.353393 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.353397 | controller |       --timeout=300s'
2026-03-14 09:44:27.353402 | controller |     - '        '
2026-03-14 09:44:27.353406 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.353410 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.353416 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.353421 | controller |     - '        '
2026-03-14 09:44:27.353425 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.353429 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.353433 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.353438 | controller |     - '        '
2026-03-14 09:44:27.353442 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.353446 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.353450 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.353454 | controller |       available"'
2026-03-14 09:44:27.353459 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.353463 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.353467 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.353471 | controller |     - '          exit 1'
2026-03-14 09:44:27.353476 | controller |     - '        fi'
2026-03-14 09:44:27.353486 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.353491 | controller |     - '        '
2026-03-14 09:44:27.353495 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.353499 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.353503 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.353508 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.353512 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.353516 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.353523 | controller |       -- \'
2026-03-14 09:44:27.353527 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.353532 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.353536 | controller |     - '    logger.go:42: 09:33:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.353540 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.353544 | controller |     - '    logger.go:42: 09:33:09 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.353548 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.353553 | controller |       found'
2026-03-14 09:44:27.353557 | controller |     - '    logger.go:42: 09:33:10 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.353561 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.353565 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.353570 | controller |     - '        '
2026-03-14 09:44:27.353574 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.353578 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.353582 | controller |     - '        '
2026-03-14 09:44:27.353586 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.353590 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.353595 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.353599 | controller |     - '          exit 1'
2026-03-14 09:44:27.353606 | controller |     - '        fi'
2026-03-14 09:44:27.353610 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.353615 | controller |     - '        '
2026-03-14 09:44:27.353619 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.353623 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.353627 | controller |       | base64 -d)'
2026-03-14 09:44:27.353631 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.353636 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.353640 | controller |     - '          exit 1'
2026-03-14 09:44:27.353644 | controller |     - '        fi'
2026-03-14 09:44:27.353648 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.353653 | controller |     - '        '
2026-03-14 09:44:27.353659 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.353663 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.353667 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.353671 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.353676 | controller |       --timeout=300s'
2026-03-14 09:44:27.353680 | controller |     - '        '
2026-03-14 09:44:27.353684 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.353688 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.353693 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.353697 | controller |     - '        '
2026-03-14 09:44:27.353701 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.353705 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.353709 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.353714 | controller |     - '        '
2026-03-14 09:44:27.353718 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.353722 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.353726 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.353730 | controller |       available"'
2026-03-14 09:44:27.353735 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.353739 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.353743 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.353748 | controller |     - '          exit 1'
2026-03-14 09:44:27.353752 | controller |     - '        fi'
2026-03-14 09:44:27.353757 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.353766 | controller |     - '        '
2026-03-14 09:44:27.353771 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.353775 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.353779 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.353784 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.353788 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.353792 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.353798 | controller |       -- \'
2026-03-14 09:44:27.353802 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.353807 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.353811 | controller |     - '    logger.go:42: 09:33:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.353815 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.353819 | controller |     - '    logger.go:42: 09:33:10 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.353823 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.353828 | controller |       found'
2026-03-14 09:44:27.353851 | controller |     - '    logger.go:42: 09:33:11 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.353856 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.353860 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.353865 | controller |     - '        '
2026-03-14 09:44:27.353869 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.353874 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.353878 | controller |     - '        '
2026-03-14 09:44:27.353882 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.353886 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.353891 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.353895 | controller |     - '          exit 1'
2026-03-14 09:44:27.353899 | controller |     - '        fi'
2026-03-14 09:44:27.353903 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.353908 | controller |     - '        '
2026-03-14 09:44:27.353912 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.353916 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.353920 | controller |       | base64 -d)'
2026-03-14 09:44:27.353924 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.353932 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.353936 | controller |     - '          exit 1'
2026-03-14 09:44:27.353940 | controller |     - '        fi'
2026-03-14 09:44:27.353944 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.353949 | controller |     - '        '
2026-03-14 09:44:27.353953 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.353957 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.353961 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.353965 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.353969 | controller |       --timeout=300s'
2026-03-14 09:44:27.353974 | controller |     - '        '
2026-03-14 09:44:27.353978 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.353982 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.353986 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.353991 | controller |     - '        '
2026-03-14 09:44:27.353995 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.353999 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.354003 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.354009 | controller |     - '        '
2026-03-14 09:44:27.354014 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.354019 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.354041 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.354046 | controller |       available"'
2026-03-14 09:44:27.354051 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.354058 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.354063 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.354068 | controller |     - '          exit 1'
2026-03-14 09:44:27.354073 | controller |     - '        fi'
2026-03-14 09:44:27.354078 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.354083 | controller |     - '        '
2026-03-14 09:44:27.354094 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.354100 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.354105 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.354109 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.354114 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.354119 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.354124 | controller |       -- \'
2026-03-14 09:44:27.354129 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.354134 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.354139 | controller |     - '    logger.go:42: 09:33:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.354144 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.354149 | controller |     - '    logger.go:42: 09:33:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.354154 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.354159 | controller |       found'
2026-03-14 09:44:27.354164 | controller |     - '    logger.go:42: 09:33:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.354169 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.354174 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.354179 | controller |     - '        '
2026-03-14 09:44:27.354184 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.354189 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.354193 | controller |     - '        '
2026-03-14 09:44:27.354198 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.354203 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.354208 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.354213 | controller |     - '          exit 1'
2026-03-14 09:44:27.354218 | controller |     - '        fi'
2026-03-14 09:44:27.354223 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.354228 | controller |     - '        '
2026-03-14 09:44:27.354233 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.354238 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.354243 | controller |       | base64 -d)'
2026-03-14 09:44:27.354248 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.354253 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.354260 | controller |     - '          exit 1'
2026-03-14 09:44:27.354265 | controller |     - '        fi'
2026-03-14 09:44:27.354270 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.354275 | controller |     - '        '
2026-03-14 09:44:27.354279 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.354283 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.354288 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.354305 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.354310 | controller |       --timeout=300s'
2026-03-14 09:44:27.354316 | controller |     - '        '
2026-03-14 09:44:27.354320 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.354324 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.354329 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.354333 | controller |     - '        '
2026-03-14 09:44:27.354337 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.354341 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.354345 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.354349 | controller |     - '        '
2026-03-14 09:44:27.354354 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.354358 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.354362 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.354366 | controller |       available"'
2026-03-14 09:44:27.354371 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.354375 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.354379 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.354383 | controller |     - '          exit 1'
2026-03-14 09:44:27.354387 | controller |     - '        fi'
2026-03-14 09:44:27.354392 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.354396 | controller |     - '        '
2026-03-14 09:44:27.354400 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.354410 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.354415 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.354419 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.354423 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.354428 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.354432 | controller |       -- \'
2026-03-14 09:44:27.354436 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.354440 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.354444 | controller |     - '    logger.go:42: 09:33:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.354449 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.354453 | controller |     - '    logger.go:42: 09:33:13 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.354459 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.354463 | controller |       found'
2026-03-14 09:44:27.354468 | controller |     - '    logger.go:42: 09:33:14 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.354472 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.354476 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.354481 | controller |     - '        '
2026-03-14 09:44:27.354487 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.354491 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.354496 | controller |     - '        '
2026-03-14 09:44:27.354500 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.354504 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.354508 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.354513 | controller |     - '          exit 1'
2026-03-14 09:44:27.354519 | controller |     - '        fi'
2026-03-14 09:44:27.354523 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.354527 | controller |     - '        '
2026-03-14 09:44:27.354531 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.354535 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.354540 | controller |       | base64 -d)'
2026-03-14 09:44:27.354544 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.354548 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.354552 | controller |     - '          exit 1'
2026-03-14 09:44:27.354557 | controller |     - '        fi'
2026-03-14 09:44:27.354561 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.354565 | controller |     - '        '
2026-03-14 09:44:27.354569 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.354573 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.354577 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.354582 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.354586 | controller |       --timeout=300s'
2026-03-14 09:44:27.354590 | controller |     - '        '
2026-03-14 09:44:27.354594 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.354598 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.354603 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.354607 | controller |     - '        '
2026-03-14 09:44:27.354611 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.354615 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.354621 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.354625 | controller |     - '        '
2026-03-14 09:44:27.354629 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.354634 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.354638 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.354642 | controller |       available"'
2026-03-14 09:44:27.354646 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.354651 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.354657 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.354661 | controller |     - '          exit 1'
2026-03-14 09:44:27.354665 | controller |     - '        fi'
2026-03-14 09:44:27.354670 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.354674 | controller |     - '        '
2026-03-14 09:44:27.354678 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.354688 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.354693 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.354697 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.354701 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.354705 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.354709 | controller |       -- \'
2026-03-14 09:44:27.354714 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.354718 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.354722 | controller |     - '    logger.go:42: 09:33:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.354726 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.354731 | controller |     - '    logger.go:42: 09:33:14 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.354735 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.354739 | controller |       found'
2026-03-14 09:44:27.354744 | controller |     - '    logger.go:42: 09:33:15 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.354748 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.354752 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.354756 | controller |     - '        '
2026-03-14 09:44:27.354760 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.354765 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.354769 | controller |     - '        '
2026-03-14 09:44:27.354773 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.354777 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.354781 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.354786 | controller |     - '          exit 1'
2026-03-14 09:44:27.354790 | controller |     - '        fi'
2026-03-14 09:44:27.354796 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.354800 | controller |     - '        '
2026-03-14 09:44:27.354805 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.354809 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.354813 | controller |       | base64 -d)'
2026-03-14 09:44:27.354817 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.354822 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.354826 | controller |     - '          exit 1'
2026-03-14 09:44:27.354830 | controller |     - '        fi'
2026-03-14 09:44:27.354834 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.354839 | controller |     - '        '
2026-03-14 09:44:27.354843 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.354847 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.354851 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.354857 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.354861 | controller |       --timeout=300s'
2026-03-14 09:44:27.354866 | controller |     - '        '
2026-03-14 09:44:27.354870 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.354874 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.354878 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.354882 | controller |     - '        '
2026-03-14 09:44:27.354887 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.354891 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.354895 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.354899 | controller |     - '        '
2026-03-14 09:44:27.354903 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.354908 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.354912 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.354916 | controller |       available"'
2026-03-14 09:44:27.354921 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.354926 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.354931 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.354936 | controller |     - '          exit 1'
2026-03-14 09:44:27.354941 | controller |     - '        fi'
2026-03-14 09:44:27.354946 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.354951 | controller |     - '        '
2026-03-14 09:44:27.354955 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.354959 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.354968 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.354974 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.354978 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.354982 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.354986 | controller |       -- \'
2026-03-14 09:44:27.354991 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.354995 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.354999 | controller |     - '    logger.go:42: 09:33:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.355003 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.355008 | controller |     - '    logger.go:42: 09:33:15 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.355012 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.355016 | controller |       found'
2026-03-14 09:44:27.355024 | controller |     - '    logger.go:42: 09:33:16 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.355028 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.355032 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.355037 | controller |     - '        '
2026-03-14 09:44:27.355041 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.355058 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.355063 | controller |     - '        '
2026-03-14 09:44:27.355067 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.355071 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.355075 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.355080 | controller |     - '          exit 1'
2026-03-14 09:44:27.355084 | controller |     - '        fi'
2026-03-14 09:44:27.355088 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.355092 | controller |     - '        '
2026-03-14 09:44:27.355097 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.355101 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.355105 | controller |       | base64 -d)'
2026-03-14 09:44:27.355109 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.355113 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.355118 | controller |     - '          exit 1'
2026-03-14 09:44:27.355122 | controller |     - '        fi'
2026-03-14 09:44:27.355126 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.355130 | controller |     - '        '
2026-03-14 09:44:27.355135 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.355139 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.355143 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.355147 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.355151 | controller |       --timeout=300s'
2026-03-14 09:44:27.355156 | controller |     - '        '
2026-03-14 09:44:27.355160 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.355164 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.355168 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.355173 | controller |     - '        '
2026-03-14 09:44:27.355177 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.355181 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.355185 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.355189 | controller |     - '        '
2026-03-14 09:44:27.355193 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.355198 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.355202 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.355206 | controller |       available"'
2026-03-14 09:44:27.355210 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.355215 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.355219 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.355223 | controller |     - '          exit 1'
2026-03-14 09:44:27.355227 | controller |     - '        fi'
2026-03-14 09:44:27.355231 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.355236 | controller |     - '        '
2026-03-14 09:44:27.355240 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.355244 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.355257 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.355262 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.355266 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.355270 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.355275 | controller |       -- \'
2026-03-14 09:44:27.355279 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.355283 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.355302 | controller |     - '    logger.go:42: 09:33:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.355309 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.355314 | controller |     - '    logger.go:42: 09:33:16 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.355318 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.355322 | controller |       found'
2026-03-14 09:44:27.355327 | controller |     - '    logger.go:42: 09:33:17 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.355331 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.355335 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.355339 | controller |     - '        '
2026-03-14 09:44:27.355343 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.355348 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.355352 | controller |     - '        '
2026-03-14 09:44:27.355356 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.355360 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.355365 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.355369 | controller |     - '          exit 1'
2026-03-14 09:44:27.355373 | controller |     - '        fi'
2026-03-14 09:44:27.355377 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.355382 | controller |     - '        '
2026-03-14 09:44:27.355387 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.355392 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.355396 | controller |       | base64 -d)'
2026-03-14 09:44:27.355400 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.355404 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.355408 | controller |     - '          exit 1'
2026-03-14 09:44:27.355413 | controller |     - '        fi'
2026-03-14 09:44:27.355417 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.355421 | controller |     - '        '
2026-03-14 09:44:27.355425 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.355430 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.355434 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.355438 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.355442 | controller |       --timeout=300s'
2026-03-14 09:44:27.355447 | controller |     - '        '
2026-03-14 09:44:27.355451 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.355455 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.355459 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.355465 | controller |     - '        '
2026-03-14 09:44:27.355470 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.355474 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.355478 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.355482 | controller |     - '        '
2026-03-14 09:44:27.355486 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.355491 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.355495 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.355499 | controller |       available"'
2026-03-14 09:44:27.355503 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.355507 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.355512 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.355516 | controller |     - '          exit 1'
2026-03-14 09:44:27.355520 | controller |     - '        fi'
2026-03-14 09:44:27.355524 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.355529 | controller |     - '        '
2026-03-14 09:44:27.355533 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.355537 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.355541 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.355551 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.355556 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.355560 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.355564 | controller |       -- \'
2026-03-14 09:44:27.355569 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.355573 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.355579 | controller |     - '    logger.go:42: 09:33:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.355583 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.355587 | controller |     - '    logger.go:42: 09:33:17 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.355592 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.355596 | controller |       found'
2026-03-14 09:44:27.355600 | controller |     - '    logger.go:42: 09:33:18 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.355605 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.355609 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.355613 | controller |     - '        '
2026-03-14 09:44:27.355617 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.355622 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.355626 | controller |     - '        '
2026-03-14 09:44:27.355631 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.355640 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.355645 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.355651 | controller |     - '          exit 1'
2026-03-14 09:44:27.355657 | controller |     - '        fi'
2026-03-14 09:44:27.355661 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.355667 | controller |     - '        '
2026-03-14 09:44:27.355672 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.355676 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.355680 | controller |       | base64 -d)'
2026-03-14 09:44:27.355684 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.355689 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.355693 | controller |     - '          exit 1'
2026-03-14 09:44:27.355697 | controller |     - '        fi'
2026-03-14 09:44:27.355701 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.355706 | controller |     - '        '
2026-03-14 09:44:27.355710 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.355714 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.355719 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.355723 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.355727 | controller |       --timeout=300s'
2026-03-14 09:44:27.355731 | controller |     - '        '
2026-03-14 09:44:27.355736 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.355740 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.355744 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.355748 | controller |     - '        '
2026-03-14 09:44:27.355752 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.355757 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.355761 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.355765 | controller |     - '        '
2026-03-14 09:44:27.355769 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.355774 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.355778 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.355782 | controller |       available"'
2026-03-14 09:44:27.355786 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.355790 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.355795 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.355799 | controller |     - '          exit 1'
2026-03-14 09:44:27.355803 | controller |     - '        fi'
2026-03-14 09:44:27.355810 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.355814 | controller |     - '        '
2026-03-14 09:44:27.355818 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.355823 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.355827 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.355837 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.355842 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.355846 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.355851 | controller |       -- \'
2026-03-14 09:44:27.355857 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.355861 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.355865 | controller |     - '    logger.go:42: 09:33:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.355870 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.355874 | controller |     - '    logger.go:42: 09:33:18 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.355878 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.355882 | controller |       found'
2026-03-14 09:44:27.355887 | controller |     - '    logger.go:42: 09:33:19 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.355891 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.355895 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.355899 | controller |     - '        '
2026-03-14 09:44:27.355903 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.355908 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.355912 | controller |     - '        '
2026-03-14 09:44:27.355916 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.355920 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.355925 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.355929 | controller |     - '          exit 1'
2026-03-14 09:44:27.355933 | controller |     - '        fi'
2026-03-14 09:44:27.355937 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.355941 | controller |     - '        '
2026-03-14 09:44:27.355946 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.355950 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.355954 | controller |       | base64 -d)'
2026-03-14 09:44:27.355960 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.355965 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.355970 | controller |     - '          exit 1'
2026-03-14 09:44:27.355974 | controller |     - '        fi'
2026-03-14 09:44:27.355979 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.355985 | controller |     - '        '
2026-03-14 09:44:27.355989 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.355993 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.355998 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.356002 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.356006 | controller |       --timeout=300s'
2026-03-14 09:44:27.356010 | controller |     - '        '
2026-03-14 09:44:27.356014 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.356019 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.356023 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.356027 | controller |     - '        '
2026-03-14 09:44:27.356031 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.356036 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.356040 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.356044 | controller |     - '        '
2026-03-14 09:44:27.356050 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.356054 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.356058 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.356063 | controller |       available"'
2026-03-14 09:44:27.356067 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.356071 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.356075 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.356079 | controller |     - '          exit 1'
2026-03-14 09:44:27.356084 | controller |     - '        fi'
2026-03-14 09:44:27.356088 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.356092 | controller |     - '        '
2026-03-14 09:44:27.356096 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.356101 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.356105 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.356109 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.356118 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.356123 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.356127 | controller |       -- \'
2026-03-14 09:44:27.356131 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.356136 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.356140 | controller |     - '    logger.go:42: 09:33:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.356144 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.356148 | controller |     - '    logger.go:42: 09:33:19 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.356153 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.356157 | controller |       found'
2026-03-14 09:44:27.356161 | controller |     - '    logger.go:42: 09:33:20 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.356165 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.356169 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.356174 | controller |     - '        '
2026-03-14 09:44:27.356178 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.356182 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.356186 | controller |     - '        '
2026-03-14 09:44:27.356190 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.356195 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.356199 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.356203 | controller |     - '          exit 1'
2026-03-14 09:44:27.356207 | controller |     - '        fi'
2026-03-14 09:44:27.356211 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.356217 | controller |     - '        '
2026-03-14 09:44:27.356222 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.356226 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.356230 | controller |       | base64 -d)'
2026-03-14 09:44:27.356234 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.356239 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.356244 | controller |     - '          exit 1'
2026-03-14 09:44:27.356249 | controller |     - '        fi'
2026-03-14 09:44:27.356253 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.356257 | controller |     - '        '
2026-03-14 09:44:27.356261 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.356266 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.356270 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.356274 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.356278 | controller |       --timeout=300s'
2026-03-14 09:44:27.356282 | controller |     - '        '
2026-03-14 09:44:27.356290 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.356307 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.356311 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.356316 | controller |     - '        '
2026-03-14 09:44:27.356320 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.356324 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.356328 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.356332 | controller |     - '        '
2026-03-14 09:44:27.356337 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.356341 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.356345 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.356349 | controller |       available"'
2026-03-14 09:44:27.356354 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.356358 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.356362 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.356366 | controller |     - '          exit 1'
2026-03-14 09:44:27.356371 | controller |     - '        fi'
2026-03-14 09:44:27.356375 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.356379 | controller |     - '        '
2026-03-14 09:44:27.356383 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.356387 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.356392 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.356396 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.356407 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.356412 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.356416 | controller |       -- \'
2026-03-14 09:44:27.356420 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.356425 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.356429 | controller |     - '    logger.go:42: 09:33:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.356433 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.356437 | controller |     - '    logger.go:42: 09:33:21 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.356446 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.356451 | controller |       found'
2026-03-14 09:44:27.356456 | controller |     - '    logger.go:42: 09:33:22 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.356462 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.356468 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.356474 | controller |     - '        '
2026-03-14 09:44:27.356480 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.356486 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.356491 | controller |     - '        '
2026-03-14 09:44:27.356497 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.356502 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.356508 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.356514 | controller |     - '          exit 1'
2026-03-14 09:44:27.356520 | controller |     - '        fi'
2026-03-14 09:44:27.356525 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.356529 | controller |     - '        '
2026-03-14 09:44:27.356533 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.356537 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.356542 | controller |       | base64 -d)'
2026-03-14 09:44:27.356546 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.356550 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.356554 | controller |     - '          exit 1'
2026-03-14 09:44:27.356559 | controller |     - '        fi'
2026-03-14 09:44:27.356563 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.356567 | controller |     - '        '
2026-03-14 09:44:27.356572 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.356577 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.356583 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.356590 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.356596 | controller |       --timeout=300s'
2026-03-14 09:44:27.356601 | controller |     - '        '
2026-03-14 09:44:27.356607 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.356612 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.356618 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.356628 | controller |     - '        '
2026-03-14 09:44:27.356636 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.356642 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.356647 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.356653 | controller |     - '        '
2026-03-14 09:44:27.356659 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.356678 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.356684 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.356690 | controller |       available"'
2026-03-14 09:44:27.356696 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.356702 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.356708 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.356716 | controller |     - '          exit 1'
2026-03-14 09:44:27.356721 | controller |     - '        fi'
2026-03-14 09:44:27.356726 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.356732 | controller |     - '        '
2026-03-14 09:44:27.356736 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.356740 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.356744 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.356749 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.356753 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.356767 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.356774 | controller |       -- \'
2026-03-14 09:44:27.356780 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.356786 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.356792 | controller |     - '    logger.go:42: 09:33:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.356798 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.356803 | controller |     - '    logger.go:42: 09:33:22 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.356809 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.356815 | controller |       found'
2026-03-14 09:44:27.356821 | controller |     - '    logger.go:42: 09:33:23 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.356825 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.356829 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.356833 | controller |     - '        '
2026-03-14 09:44:27.356838 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.356842 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.356846 | controller |     - '        '
2026-03-14 09:44:27.356850 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.356855 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.356859 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.356863 | controller |     - '          exit 1'
2026-03-14 09:44:27.356867 | controller |     - '        fi'
2026-03-14 09:44:27.356872 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.356876 | controller |     - '        '
2026-03-14 09:44:27.356880 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.356884 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.356889 | controller |       | base64 -d)'
2026-03-14 09:44:27.356893 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.356897 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.356901 | controller |     - '          exit 1'
2026-03-14 09:44:27.356905 | controller |     - '        fi'
2026-03-14 09:44:27.356910 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.356914 | controller |     - '        '
2026-03-14 09:44:27.356918 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.356922 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.356929 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.356934 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.356940 | controller |       --timeout=300s'
2026-03-14 09:44:27.356944 | controller |     - '        '
2026-03-14 09:44:27.356950 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.356954 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.356958 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.356962 | controller |     - '        '
2026-03-14 09:44:27.356967 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.356971 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.356975 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.356979 | controller |     - '        '
2026-03-14 09:44:27.356984 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.356988 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.356992 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.356996 | controller |       available"'
2026-03-14 09:44:27.357000 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.357005 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.357009 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.357013 | controller |     - '          exit 1'
2026-03-14 09:44:27.357017 | controller |     - '        fi'
2026-03-14 09:44:27.357021 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.357025 | controller |     - '        '
2026-03-14 09:44:27.357030 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.357034 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.357038 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.357042 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.357046 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.357057 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.357062 | controller |       -- \'
2026-03-14 09:44:27.357066 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.357071 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.357075 | controller |     - '    logger.go:42: 09:33:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.357079 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.357083 | controller |     - '    logger.go:42: 09:33:23 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.357087 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.357092 | controller |       found'
2026-03-14 09:44:27.357096 | controller |     - '    logger.go:42: 09:33:24 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.357100 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.357104 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.357109 | controller |     - '        '
2026-03-14 09:44:27.357113 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.357117 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.357123 | controller |     - '        '
2026-03-14 09:44:27.357127 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.357131 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.357136 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.357140 | controller |     - '          exit 1'
2026-03-14 09:44:27.357144 | controller |     - '        fi'
2026-03-14 09:44:27.357148 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.357152 | controller |     - '        '
2026-03-14 09:44:27.357157 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.357161 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.357165 | controller |       | base64 -d)'
2026-03-14 09:44:27.357169 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.357174 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.357178 | controller |     - '          exit 1'
2026-03-14 09:44:27.357182 | controller |     - '        fi'
2026-03-14 09:44:27.357186 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.357190 | controller |     - '        '
2026-03-14 09:44:27.357195 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.357199 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.357203 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.357207 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.357211 | controller |       --timeout=300s'
2026-03-14 09:44:27.357216 | controller |     - '        '
2026-03-14 09:44:27.357220 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.357224 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.357228 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.357232 | controller |     - '        '
2026-03-14 09:44:27.357237 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.357241 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.357245 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.357249 | controller |     - '        '
2026-03-14 09:44:27.357253 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.357258 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.357262 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.357266 | controller |       available"'
2026-03-14 09:44:27.357270 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.357274 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.357279 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.357283 | controller |     - '          exit 1'
2026-03-14 09:44:27.357290 | controller |     - '        fi'
2026-03-14 09:44:27.357341 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.357346 | controller |     - '        '
2026-03-14 09:44:27.357350 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.357354 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.357358 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.357365 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.357369 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.357373 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.357378 | controller |       -- \'
2026-03-14 09:44:27.357389 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.357394 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.357398 | controller |     - '    logger.go:42: 09:33:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.357402 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.357406 | controller |     - '    logger.go:42: 09:33:24 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.357410 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.357415 | controller |       found'
2026-03-14 09:44:27.357419 | controller |     - '    logger.go:42: 09:33:25 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.357423 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.357427 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.357431 | controller |     - '        '
2026-03-14 09:44:27.357436 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.357440 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.357444 | controller |     - '        '
2026-03-14 09:44:27.357448 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.357452 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.357457 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.357461 | controller |     - '          exit 1'
2026-03-14 09:44:27.357465 | controller |     - '        fi'
2026-03-14 09:44:27.357469 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.357473 | controller |     - '        '
2026-03-14 09:44:27.357478 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.357482 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.357486 | controller |       | base64 -d)'
2026-03-14 09:44:27.357490 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.357494 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.357499 | controller |     - '          exit 1'
2026-03-14 09:44:27.357503 | controller |     - '        fi'
2026-03-14 09:44:27.357507 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.357511 | controller |     - '        '
2026-03-14 09:44:27.357515 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.357520 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.357524 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.357530 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.357534 | controller |       --timeout=300s'
2026-03-14 09:44:27.357539 | controller |     - '        '
2026-03-14 09:44:27.357543 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.357547 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.357551 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.357560 | controller |     - '        '
2026-03-14 09:44:27.357566 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.357572 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.357578 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.357584 | controller |     - '        '
2026-03-14 09:44:27.357590 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.357596 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.357601 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.357607 | controller |       available"'
2026-03-14 09:44:27.357612 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.357618 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.357624 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.357630 | controller |     - '          exit 1'
2026-03-14 09:44:27.357636 | controller |     - '        fi'
2026-03-14 09:44:27.357641 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.357647 | controller |     - '        '
2026-03-14 09:44:27.357653 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.357658 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.357662 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.357666 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.357671 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.357675 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.357679 | controller |       -- \'
2026-03-14 09:44:27.357691 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.357695 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.357700 | controller |     - '    logger.go:42: 09:33:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.357706 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.357711 | controller |     - '    logger.go:42: 09:33:25 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.357717 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.357723 | controller |       found'
2026-03-14 09:44:27.357729 | controller |     - '    logger.go:42: 09:33:26 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.357734 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.357740 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.357746 | controller |     - '        '
2026-03-14 09:44:27.357752 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.357758 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.357762 | controller |     - '        '
2026-03-14 09:44:27.357766 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.357771 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.357775 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.357779 | controller |     - '          exit 1'
2026-03-14 09:44:27.357783 | controller |     - '        fi'
2026-03-14 09:44:27.357787 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.357794 | controller |     - '        '
2026-03-14 09:44:27.357799 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.357803 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.357807 | controller |       | base64 -d)'
2026-03-14 09:44:27.357811 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.357816 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.357820 | controller |     - '          exit 1'
2026-03-14 09:44:27.357824 | controller |     - '        fi'
2026-03-14 09:44:27.357828 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.357832 | controller |     - '        '
2026-03-14 09:44:27.357836 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.357841 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.357845 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.357849 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.357853 | controller |       --timeout=300s'
2026-03-14 09:44:27.357857 | controller |     - '        '
2026-03-14 09:44:27.357862 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.357866 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.357873 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.357877 | controller |     - '        '
2026-03-14 09:44:27.357881 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.357885 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.357890 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.357894 | controller |     - '        '
2026-03-14 09:44:27.357898 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.357902 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.357906 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.357911 | controller |       available"'
2026-03-14 09:44:27.357915 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.357919 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.357923 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.357927 | controller |     - '          exit 1'
2026-03-14 09:44:27.357932 | controller |     - '        fi'
2026-03-14 09:44:27.357936 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.357940 | controller |     - '        '
2026-03-14 09:44:27.357944 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.357948 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.357953 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.357957 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.357961 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.357965 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.357969 | controller |       -- \'
2026-03-14 09:44:27.357973 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.357986 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.357991 | controller |     - '    logger.go:42: 09:33:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.357995 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.357999 | controller |     - '    logger.go:42: 09:33:26 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.358004 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.358008 | controller |       found'
2026-03-14 09:44:27.358012 | controller |     - '    logger.go:42: 09:33:27 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.358016 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.358020 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.358025 | controller |     - '        '
2026-03-14 09:44:27.358045 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.358051 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.358055 | controller |     - '        '
2026-03-14 09:44:27.358059 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.358063 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.358071 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.358075 | controller |     - '          exit 1'
2026-03-14 09:44:27.358080 | controller |     - '        fi'
2026-03-14 09:44:27.358084 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.358088 | controller |     - '        '
2026-03-14 09:44:27.358093 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.358098 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.358103 | controller |       | base64 -d)'
2026-03-14 09:44:27.358110 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.358115 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.358120 | controller |     - '          exit 1'
2026-03-14 09:44:27.358125 | controller |     - '        fi'
2026-03-14 09:44:27.358129 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.358133 | controller |     - '        '
2026-03-14 09:44:27.358137 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.358142 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.358146 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.358150 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.358154 | controller |       --timeout=300s'
2026-03-14 09:44:27.358159 | controller |     - '        '
2026-03-14 09:44:27.358163 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.358167 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.358171 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.358176 | controller |     - '        '
2026-03-14 09:44:27.358180 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.358184 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.358188 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.358192 | controller |     - '        '
2026-03-14 09:44:27.358197 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.358204 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.358209 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.358213 | controller |       available"'
2026-03-14 09:44:27.358217 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.358221 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.358225 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.358230 | controller |     - '          exit 1'
2026-03-14 09:44:27.358234 | controller |     - '        fi'
2026-03-14 09:44:27.358238 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.358242 | controller |     - '        '
2026-03-14 09:44:27.358247 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.358251 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.358257 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.358262 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.358268 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.358274 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.358280 | controller |       -- \'
2026-03-14 09:44:27.358286 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.358316 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.358323 | controller |     - '    logger.go:42: 09:33:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.358327 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.358332 | controller |     - '    logger.go:42: 09:33:27 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.358336 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.358340 | controller |       found'
2026-03-14 09:44:27.358344 | controller |     - '    logger.go:42: 09:33:28 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.358349 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.358353 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.358357 | controller |     - '        '
2026-03-14 09:44:27.358362 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.358366 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.358370 | controller |     - '        '
2026-03-14 09:44:27.358374 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.358379 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.358383 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.358387 | controller |     - '          exit 1'
2026-03-14 09:44:27.358391 | controller |     - '        fi'
2026-03-14 09:44:27.358396 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.358400 | controller |     - '        '
2026-03-14 09:44:27.358404 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.358408 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.358412 | controller |       | base64 -d)'
2026-03-14 09:44:27.358417 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.358421 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.358425 | controller |     - '          exit 1'
2026-03-14 09:44:27.358429 | controller |     - '        fi'
2026-03-14 09:44:27.358436 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.358440 | controller |     - '        '
2026-03-14 09:44:27.358444 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.358448 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.358453 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.358457 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.358461 | controller |       --timeout=300s'
2026-03-14 09:44:27.358468 | controller |     - '        '
2026-03-14 09:44:27.358472 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.358476 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.358480 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.358484 | controller |     - '        '
2026-03-14 09:44:27.358489 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.358493 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.358497 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.358501 | controller |     - '        '
2026-03-14 09:44:27.358505 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.358510 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.358514 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.358518 | controller |       available"'
2026-03-14 09:44:27.358522 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.358526 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.358530 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.358535 | controller |     - '          exit 1'
2026-03-14 09:44:27.358539 | controller |     - '        fi'
2026-03-14 09:44:27.358543 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.358547 | controller |     - '        '
2026-03-14 09:44:27.358552 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.358556 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.358560 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.358564 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.358568 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.358573 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.358577 | controller |       -- \'
2026-03-14 09:44:27.358581 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.358585 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.358595 | controller |     - '    logger.go:42: 09:33:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.358600 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.358604 | controller |     - '    logger.go:42: 09:33:28 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.358608 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.358615 | controller |       found'
2026-03-14 09:44:27.358619 | controller |     - '    logger.go:42: 09:33:29 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.358623 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.358627 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.358631 | controller |     - '        '
2026-03-14 09:44:27.358636 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.358641 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.358646 | controller |     - '        '
2026-03-14 09:44:27.358652 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.358658 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.358664 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.358668 | controller |     - '          exit 1'
2026-03-14 09:44:27.358672 | controller |     - '        fi'
2026-03-14 09:44:27.358677 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.358681 | controller |     - '        '
2026-03-14 09:44:27.358685 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.358689 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.358693 | controller |       | base64 -d)'
2026-03-14 09:44:27.358698 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.358702 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.358706 | controller |     - '          exit 1'
2026-03-14 09:44:27.358710 | controller |     - '        fi'
2026-03-14 09:44:27.358714 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.358718 | controller |     - '        '
2026-03-14 09:44:27.358723 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.358727 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.358731 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.358735 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.358739 | controller |       --timeout=300s'
2026-03-14 09:44:27.358744 | controller |     - '        '
2026-03-14 09:44:27.358748 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.358752 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.358756 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.358760 | controller |     - '        '
2026-03-14 09:44:27.358765 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.358769 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.358773 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.358777 | controller |     - '        '
2026-03-14 09:44:27.358781 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.358785 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.358792 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.358796 | controller |       available"'
2026-03-14 09:44:27.358800 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.358804 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.358809 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.358815 | controller |     - '          exit 1'
2026-03-14 09:44:27.358819 | controller |     - '        fi'
2026-03-14 09:44:27.358823 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.358827 | controller |     - '        '
2026-03-14 09:44:27.358831 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.358836 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.358840 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.358844 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.358850 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.358854 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.358858 | controller |       -- \'
2026-03-14 09:44:27.358863 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.358867 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.358877 | controller |     - '    logger.go:42: 09:33:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.358882 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.358886 | controller |     - '    logger.go:42: 09:33:30 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.358890 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.358894 | controller |       found'
2026-03-14 09:44:27.358899 | controller |     - '    logger.go:42: 09:33:31 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.358903 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.358907 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.358911 | controller |     - '        '
2026-03-14 09:44:27.358916 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.358920 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.358924 | controller |     - '        '
2026-03-14 09:44:27.358928 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.358932 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.358937 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.358941 | controller |     - '          exit 1'
2026-03-14 09:44:27.358945 | controller |     - '        fi'
2026-03-14 09:44:27.358950 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.358954 | controller |     - '        '
2026-03-14 09:44:27.358959 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.358964 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.358969 | controller |       | base64 -d)'
2026-03-14 09:44:27.358973 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.358978 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.358984 | controller |     - '          exit 1'
2026-03-14 09:44:27.358988 | controller |     - '        fi'
2026-03-14 09:44:27.358993 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.358997 | controller |     - '        '
2026-03-14 09:44:27.359001 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.359005 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.359009 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.359014 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.359020 | controller |       --timeout=300s'
2026-03-14 09:44:27.359024 | controller |     - '        '
2026-03-14 09:44:27.359028 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.359032 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.359037 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.359041 | controller |     - '        '
2026-03-14 09:44:27.359045 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.359049 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.359053 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.359058 | controller |     - '        '
2026-03-14 09:44:27.359062 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.359066 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.359070 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.359074 | controller |       available"'
2026-03-14 09:44:27.359079 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.359083 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.359087 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.359091 | controller |     - '          exit 1'
2026-03-14 09:44:27.359095 | controller |     - '        fi'
2026-03-14 09:44:27.359100 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.359104 | controller |     - '        '
2026-03-14 09:44:27.359110 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.359115 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.359119 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.359123 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.359127 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.359132 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.359136 | controller |       -- \'
2026-03-14 09:44:27.359140 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.359144 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.359149 | controller |     - '    logger.go:42: 09:33:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.359158 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.359163 | controller |     - '    logger.go:42: 09:33:31 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.359167 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.359171 | controller |       found'
2026-03-14 09:44:27.359176 | controller |     - '    logger.go:42: 09:33:32 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.359180 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.359184 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.359188 | controller |     - '        '
2026-03-14 09:44:27.359194 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.359198 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.359204 | controller |     - '        '
2026-03-14 09:44:27.359209 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.359213 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.359217 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.359221 | controller |     - '          exit 1'
2026-03-14 09:44:27.359226 | controller |     - '        fi'
2026-03-14 09:44:27.359230 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.359234 | controller |     - '        '
2026-03-14 09:44:27.359238 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.359242 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.359247 | controller |       | base64 -d)'
2026-03-14 09:44:27.359251 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.359255 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.359259 | controller |     - '          exit 1'
2026-03-14 09:44:27.359264 | controller |     - '        fi'
2026-03-14 09:44:27.359268 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.359272 | controller |     - '        '
2026-03-14 09:44:27.359276 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.359281 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.359285 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.359303 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.359310 | controller |       --timeout=300s'
2026-03-14 09:44:27.359314 | controller |     - '        '
2026-03-14 09:44:27.359319 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.359323 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.359327 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.359331 | controller |     - '        '
2026-03-14 09:44:27.359335 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.359340 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.359344 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.359348 | controller |     - '        '
2026-03-14 09:44:27.359352 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.359356 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.359361 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.359365 | controller |       available"'
2026-03-14 09:44:27.359369 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.359376 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.359380 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.359384 | controller |     - '          exit 1'
2026-03-14 09:44:27.359389 | controller |     - '        fi'
2026-03-14 09:44:27.359396 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.359400 | controller |     - '        '
2026-03-14 09:44:27.359404 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.359408 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.359413 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.359419 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.359423 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.359428 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.359432 | controller |       -- \'
2026-03-14 09:44:27.359436 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.359440 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.359446 | controller |     - '    logger.go:42: 09:33:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.359450 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.359461 | controller |     - '    logger.go:42: 09:33:32 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.359465 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.359469 | controller |       found'
2026-03-14 09:44:27.359474 | controller |     - '    logger.go:42: 09:33:33 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.359478 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.359482 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.359486 | controller |     - '        '
2026-03-14 09:44:27.359490 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.359495 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.359499 | controller |     - '        '
2026-03-14 09:44:27.359503 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.359507 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.359512 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.359517 | controller |     - '          exit 1'
2026-03-14 09:44:27.359523 | controller |     - '        fi'
2026-03-14 09:44:27.359529 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.359535 | controller |     - '        '
2026-03-14 09:44:27.359541 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.359547 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.359551 | controller |       | base64 -d)'
2026-03-14 09:44:27.359555 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.359559 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.359564 | controller |     - '          exit 1'
2026-03-14 09:44:27.359568 | controller |     - '        fi'
2026-03-14 09:44:27.359572 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.359576 | controller |     - '        '
2026-03-14 09:44:27.359580 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.359585 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.359589 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.359593 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.359597 | controller |       --timeout=300s'
2026-03-14 09:44:27.359601 | controller |     - '        '
2026-03-14 09:44:27.359606 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.359610 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.359614 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.359618 | controller |     - '        '
2026-03-14 09:44:27.359622 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.359629 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.359633 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.359637 | controller |     - '        '
2026-03-14 09:44:27.359642 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.359646 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.359650 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.359654 | controller |       available"'
2026-03-14 09:44:27.359658 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.359663 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.359667 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.359671 | controller |     - '          exit 1'
2026-03-14 09:44:27.359675 | controller |     - '        fi'
2026-03-14 09:44:27.359679 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.359684 | controller |     - '        '
2026-03-14 09:44:27.359688 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.359692 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.359696 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.359700 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.359705 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.359709 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.359713 | controller |       -- \'
2026-03-14 09:44:27.359717 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.359721 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.359726 | controller |     - '    logger.go:42: 09:33:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.359730 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.359734 | controller |     - '    logger.go:42: 09:33:33 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.359745 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.359749 | controller |       found'
2026-03-14 09:44:27.359753 | controller |     - '    logger.go:42: 09:33:34 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.359758 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.359762 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.359766 | controller |     - '        '
2026-03-14 09:44:27.359770 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.359775 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.359779 | controller |     - '        '
2026-03-14 09:44:27.359783 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.359787 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.359791 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.359796 | controller |     - '          exit 1'
2026-03-14 09:44:27.359800 | controller |     - '        fi'
2026-03-14 09:44:27.359804 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.359808 | controller |     - '        '
2026-03-14 09:44:27.359812 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.359818 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.359823 | controller |       | base64 -d)'
2026-03-14 09:44:27.359827 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.359831 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.359835 | controller |     - '          exit 1'
2026-03-14 09:44:27.359839 | controller |     - '        fi'
2026-03-14 09:44:27.359844 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.359848 | controller |     - '        '
2026-03-14 09:44:27.359852 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.359856 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.359860 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.359865 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.359869 | controller |       --timeout=300s'
2026-03-14 09:44:27.359873 | controller |     - '        '
2026-03-14 09:44:27.359877 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.359881 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.359886 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.359890 | controller |     - '        '
2026-03-14 09:44:27.359894 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.359901 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.359905 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.359909 | controller |     - '        '
2026-03-14 09:44:27.359913 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.359918 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.359922 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.359926 | controller |       available"'
2026-03-14 09:44:27.359930 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.359934 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.359939 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.359943 | controller |     - '          exit 1'
2026-03-14 09:44:27.359947 | controller |     - '        fi'
2026-03-14 09:44:27.359951 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.359955 | controller |     - '        '
2026-03-14 09:44:27.359961 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.359966 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.359970 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.359974 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.359978 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.359982 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.359987 | controller |       -- \'
2026-03-14 09:44:27.359991 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.359997 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.360001 | controller |     - '    logger.go:42: 09:33:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.360005 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.360009 | controller |     - '    logger.go:42: 09:33:34 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.360019 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.360024 | controller |       found'
2026-03-14 09:44:27.360028 | controller |     - '    logger.go:42: 09:33:35 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.360032 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.360037 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.360041 | controller |     - '        '
2026-03-14 09:44:27.360045 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.360049 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.360053 | controller |     - '        '
2026-03-14 09:44:27.360058 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.360062 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.360066 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.360070 | controller |     - '          exit 1'
2026-03-14 09:44:27.360075 | controller |     - '        fi'
2026-03-14 09:44:27.360079 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.360083 | controller |     - '        '
2026-03-14 09:44:27.360087 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.360092 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.360096 | controller |       | base64 -d)'
2026-03-14 09:44:27.360100 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.360104 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.360108 | controller |     - '          exit 1'
2026-03-14 09:44:27.360113 | controller |     - '        fi'
2026-03-14 09:44:27.360117 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.360121 | controller |     - '        '
2026-03-14 09:44:27.360125 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.360130 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.360134 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.360138 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.360142 | controller |       --timeout=300s'
2026-03-14 09:44:27.360147 | controller |     - '        '
2026-03-14 09:44:27.360151 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.360155 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.360159 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.360163 | controller |     - '        '
2026-03-14 09:44:27.360168 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.360172 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.360176 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.360180 | controller |     - '        '
2026-03-14 09:44:27.360187 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.360191 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.360197 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.360201 | controller |       available"'
2026-03-14 09:44:27.360205 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.360210 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.360214 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.360220 | controller |     - '          exit 1'
2026-03-14 09:44:27.360225 | controller |     - '        fi'
2026-03-14 09:44:27.360231 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.360236 | controller |     - '        '
2026-03-14 09:44:27.360242 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.360248 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.360325 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.360332 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.360338 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.360343 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.360349 | controller |       -- \'
2026-03-14 09:44:27.360354 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.360360 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.360364 | controller |     - '    logger.go:42: 09:33:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.360368 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.360373 | controller |     - '    logger.go:42: 09:33:35 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.360377 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.360382 | controller |       found'
2026-03-14 09:44:27.360396 | controller |     - '    logger.go:42: 09:33:36 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.360400 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.360405 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.360409 | controller |     - '        '
2026-03-14 09:44:27.360413 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.360417 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.360421 | controller |     - '        '
2026-03-14 09:44:27.360426 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.360430 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.360434 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.360438 | controller |     - '          exit 1'
2026-03-14 09:44:27.360443 | controller |     - '        fi'
2026-03-14 09:44:27.360447 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.360451 | controller |     - '        '
2026-03-14 09:44:27.360455 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.360459 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.360464 | controller |       | base64 -d)'
2026-03-14 09:44:27.360468 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.360472 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.360476 | controller |     - '          exit 1'
2026-03-14 09:44:27.360480 | controller |     - '        fi'
2026-03-14 09:44:27.360484 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.360492 | controller |     - '        '
2026-03-14 09:44:27.360497 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.360503 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.360509 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.360547 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.360553 | controller |       --timeout=300s'
2026-03-14 09:44:27.360559 | controller |     - '        '
2026-03-14 09:44:27.360564 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.360570 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.360575 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.360581 | controller |     - '        '
2026-03-14 09:44:27.360586 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.360591 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.360595 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.360600 | controller |     - '        '
2026-03-14 09:44:27.360604 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.360608 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.360612 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.360616 | controller |       available"'
2026-03-14 09:44:27.360621 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.360625 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.360629 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.360633 | controller |     - '          exit 1'
2026-03-14 09:44:27.360637 | controller |     - '        fi'
2026-03-14 09:44:27.360641 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.360646 | controller |     - '        '
2026-03-14 09:44:27.360650 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.360654 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.360658 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.360665 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.360670 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.360674 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.360679 | controller |       -- \'
2026-03-14 09:44:27.360683 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.360687 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.360691 | controller |     - '    logger.go:42: 09:33:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.360695 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.360700 | controller |     - '    logger.go:42: 09:33:36 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.360704 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.360708 | controller |       found'
2026-03-14 09:44:27.360722 | controller |     - '    logger.go:42: 09:33:37 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.360727 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.360731 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.360735 | controller |     - '        '
2026-03-14 09:44:27.360739 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.360744 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.360748 | controller |     - '        '
2026-03-14 09:44:27.360752 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.360756 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.360760 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.360765 | controller |     - '          exit 1'
2026-03-14 09:44:27.360769 | controller |     - '        fi'
2026-03-14 09:44:27.360773 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.360777 | controller |     - '        '
2026-03-14 09:44:27.360784 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.360788 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.360792 | controller |       | base64 -d)'
2026-03-14 09:44:27.360796 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.360801 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.360805 | controller |     - '          exit 1'
2026-03-14 09:44:27.360809 | controller |     - '        fi'
2026-03-14 09:44:27.360813 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.360818 | controller |     - '        '
2026-03-14 09:44:27.360824 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.360829 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.360835 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.360840 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.360846 | controller |       --timeout=300s'
2026-03-14 09:44:27.360851 | controller |     - '        '
2026-03-14 09:44:27.360856 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.360860 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.360864 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.360869 | controller |     - '        '
2026-03-14 09:44:27.360875 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.360879 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.360883 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.360887 | controller |     - '        '
2026-03-14 09:44:27.360891 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.360896 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.360900 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.360904 | controller |       available"'
2026-03-14 09:44:27.360908 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.360913 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.360917 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.360922 | controller |     - '          exit 1'
2026-03-14 09:44:27.360931 | controller |     - '        fi'
2026-03-14 09:44:27.360936 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.360941 | controller |     - '        '
2026-03-14 09:44:27.360946 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.360951 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.360955 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.360959 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.360963 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.360967 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.360974 | controller |       -- \'
2026-03-14 09:44:27.360978 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.360983 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.360987 | controller |     - '    logger.go:42: 09:33:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.360991 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.360995 | controller |     - '    logger.go:42: 09:33:38 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.360999 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.361004 | controller |       found'
2026-03-14 09:44:27.361008 | controller |     - '    logger.go:42: 09:33:39 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.361018 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.361023 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.361027 | controller |     - '        '
2026-03-14 09:44:27.361031 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.361036 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.361040 | controller |     - '        '
2026-03-14 09:44:27.361044 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.361048 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.361053 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.361057 | controller |     - '          exit 1'
2026-03-14 09:44:27.361061 | controller |     - '        fi'
2026-03-14 09:44:27.361065 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.361069 | controller |     - '        '
2026-03-14 09:44:27.361074 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.361078 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.361082 | controller |       | base64 -d)'
2026-03-14 09:44:27.361086 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.361091 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.361095 | controller |     - '          exit 1'
2026-03-14 09:44:27.361099 | controller |     - '        fi'
2026-03-14 09:44:27.361103 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.361107 | controller |     - '        '
2026-03-14 09:44:27.361111 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.361116 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.361120 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.361124 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.361128 | controller |       --timeout=300s'
2026-03-14 09:44:27.361135 | controller |     - '        '
2026-03-14 09:44:27.361139 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.361143 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.361147 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.361151 | controller |     - '        '
2026-03-14 09:44:27.361156 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.361160 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.361164 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.361168 | controller |     - '        '
2026-03-14 09:44:27.361173 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.361177 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.361181 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.361185 | controller |       available"'
2026-03-14 09:44:27.361189 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.361194 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.361198 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.361202 | controller |     - '          exit 1'
2026-03-14 09:44:27.361206 | controller |     - '        fi'
2026-03-14 09:44:27.361211 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.361215 | controller |     - '        '
2026-03-14 09:44:27.361219 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.361223 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.361227 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.361232 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.361236 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.361240 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.361244 | controller |       -- \'
2026-03-14 09:44:27.361248 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.361254 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.361259 | controller |     - '    logger.go:42: 09:33:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.361263 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.361267 | controller |     - '    logger.go:42: 09:33:39 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.361271 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.361275 | controller |       found'
2026-03-14 09:44:27.361280 | controller |     - '    logger.go:42: 09:33:40 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.361284 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.361307 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.361317 | controller |     - '        '
2026-03-14 09:44:27.361322 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.361326 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.361330 | controller |     - '        '
2026-03-14 09:44:27.361335 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.361341 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.361345 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.361350 | controller |     - '          exit 1'
2026-03-14 09:44:27.361354 | controller |     - '        fi'
2026-03-14 09:44:27.361358 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.361362 | controller |     - '        '
2026-03-14 09:44:27.361367 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.361371 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.361375 | controller |       | base64 -d)'
2026-03-14 09:44:27.361379 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.361384 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.361388 | controller |     - '          exit 1'
2026-03-14 09:44:27.361392 | controller |     - '        fi'
2026-03-14 09:44:27.361398 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.361403 | controller |     - '        '
2026-03-14 09:44:27.361407 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.361411 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.361415 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.361420 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.361424 | controller |       --timeout=300s'
2026-03-14 09:44:27.361428 | controller |     - '        '
2026-03-14 09:44:27.361432 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.361437 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.361441 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.361445 | controller |     - '        '
2026-03-14 09:44:27.361449 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.361454 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.361458 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.361462 | controller |     - '        '
2026-03-14 09:44:27.361466 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.361470 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.361475 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.361479 | controller |       available"'
2026-03-14 09:44:27.361483 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.361487 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.361492 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.361496 | controller |     - '          exit 1'
2026-03-14 09:44:27.361500 | controller |     - '        fi'
2026-03-14 09:44:27.361504 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.361509 | controller |     - '        '
2026-03-14 09:44:27.361513 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.361517 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.361521 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.361525 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.361531 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.361535 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.361540 | controller |       -- \'
2026-03-14 09:44:27.361544 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.361548 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.361552 | controller |     - '    logger.go:42: 09:33:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.361557 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.361561 | controller |     - '    logger.go:42: 09:33:40 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.361565 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.361569 | controller |       found'
2026-03-14 09:44:27.361574 | controller |     - '    logger.go:42: 09:33:41 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.361578 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.361584 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.361588 | controller |     - '        '
2026-03-14 09:44:27.361598 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.361602 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.361607 | controller |     - '        '
2026-03-14 09:44:27.361611 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.361615 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.361619 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.361623 | controller |     - '          exit 1'
2026-03-14 09:44:27.361628 | controller |     - '        fi'
2026-03-14 09:44:27.361632 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.361636 | controller |     - '        '
2026-03-14 09:44:27.361640 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.361644 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.361648 | controller |       | base64 -d)'
2026-03-14 09:44:27.361653 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.361657 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.361661 | controller |     - '          exit 1'
2026-03-14 09:44:27.361665 | controller |     - '        fi'
2026-03-14 09:44:27.361670 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.361674 | controller |     - '        '
2026-03-14 09:44:27.361678 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.361682 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.361686 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.361691 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.361695 | controller |       --timeout=300s'
2026-03-14 09:44:27.361699 | controller |     - '        '
2026-03-14 09:44:27.361703 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.361707 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.361712 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.361716 | controller |     - '        '
2026-03-14 09:44:27.361720 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.361726 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.361730 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.361734 | controller |     - '        '
2026-03-14 09:44:27.361739 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.361743 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.361747 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.361751 | controller |       available"'
2026-03-14 09:44:27.361756 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.361760 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.361764 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.361768 | controller |     - '          exit 1'
2026-03-14 09:44:27.361772 | controller |     - '        fi'
2026-03-14 09:44:27.361777 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.361781 | controller |     - '        '
2026-03-14 09:44:27.361785 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.361789 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.361793 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.361798 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.361802 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.361806 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.361810 | controller |       -- \'
2026-03-14 09:44:27.361815 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.361819 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.361823 | controller |     - '    logger.go:42: 09:33:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.361827 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.361833 | controller |     - '    logger.go:42: 09:33:41 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.361837 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.361842 | controller |       found'
2026-03-14 09:44:27.361846 | controller |     - '    logger.go:42: 09:33:42 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.361850 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.361856 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.361861 | controller |     - '        '
2026-03-14 09:44:27.361865 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.361875 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.361880 | controller |     - '        '
2026-03-14 09:44:27.361884 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.361888 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.361893 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.361898 | controller |     - '          exit 1'
2026-03-14 09:44:27.361904 | controller |     - '        fi'
2026-03-14 09:44:27.361910 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.361915 | controller |     - '        '
2026-03-14 09:44:27.361921 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.361928 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.361934 | controller |       | base64 -d)'
2026-03-14 09:44:27.361940 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.361946 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.361951 | controller |     - '          exit 1'
2026-03-14 09:44:27.361958 | controller |     - '        fi'
2026-03-14 09:44:27.361964 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.361969 | controller |     - '        '
2026-03-14 09:44:27.361973 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.361978 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.361983 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.361988 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.361992 | controller |       --timeout=300s'
2026-03-14 09:44:27.361996 | controller |     - '        '
2026-03-14 09:44:27.362001 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.362005 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.362009 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.362013 | controller |     - '        '
2026-03-14 09:44:27.362017 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.362022 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.362028 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.362033 | controller |     - '        '
2026-03-14 09:44:27.362072 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.362078 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.362082 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.362086 | controller |       available"'
2026-03-14 09:44:27.362090 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.362097 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.362101 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.362105 | controller |     - '          exit 1'
2026-03-14 09:44:27.362109 | controller |     - '        fi'
2026-03-14 09:44:27.362114 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.362118 | controller |     - '        '
2026-03-14 09:44:27.362122 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.362126 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.362130 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.362135 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.362139 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.362143 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.362147 | controller |       -- \'
2026-03-14 09:44:27.362151 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.362156 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.362162 | controller |     - '    logger.go:42: 09:33:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.362166 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.362170 | controller |     - '    logger.go:42: 09:33:42 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.362175 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.362179 | controller |       found'
2026-03-14 09:44:27.362183 | controller |     - '    logger.go:42: 09:33:43 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.362187 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.362192 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.362196 | controller |     - '        '
2026-03-14 09:44:27.362200 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.362211 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.362216 | controller |     - '        '
2026-03-14 09:44:27.362220 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.362224 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.362229 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.362233 | controller |     - '          exit 1'
2026-03-14 09:44:27.362238 | controller |     - '        fi'
2026-03-14 09:44:27.362243 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.362248 | controller |     - '        '
2026-03-14 09:44:27.362254 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.362259 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.362264 | controller |       | base64 -d)'
2026-03-14 09:44:27.362270 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.362280 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.362285 | controller |     - '          exit 1'
2026-03-14 09:44:27.362309 | controller |     - '        fi'
2026-03-14 09:44:27.362316 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.362322 | controller |     - '        '
2026-03-14 09:44:27.362327 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.362332 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.362337 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.362342 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.362348 | controller |       --timeout=300s'
2026-03-14 09:44:27.362353 | controller |     - '        '
2026-03-14 09:44:27.362359 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.362365 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.362369 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.362374 | controller |     - '        '
2026-03-14 09:44:27.362378 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.362382 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.362386 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.362390 | controller |     - '        '
2026-03-14 09:44:27.362394 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.362399 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.362405 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.362410 | controller |       available"'
2026-03-14 09:44:27.362414 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.362418 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.362422 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.362426 | controller |     - '          exit 1'
2026-03-14 09:44:27.362431 | controller |     - '        fi'
2026-03-14 09:44:27.362435 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.362439 | controller |     - '        '
2026-03-14 09:44:27.362443 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.362447 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.362452 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.362456 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.362460 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.362464 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.362468 | controller |       -- \'
2026-03-14 09:44:27.362473 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.362477 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.362481 | controller |     - '    logger.go:42: 09:33:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.362485 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.362490 | controller |     - '    logger.go:42: 09:33:43 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.362496 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.362501 | controller |       found'
2026-03-14 09:44:27.362505 | controller |     - '    logger.go:42: 09:33:44 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.362509 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.362513 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.362517 | controller |     - '        '
2026-03-14 09:44:27.362521 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.362526 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.362530 | controller |     - '        '
2026-03-14 09:44:27.362542 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.362547 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.362551 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.362555 | controller |     - '          exit 1'
2026-03-14 09:44:27.362562 | controller |     - '        fi'
2026-03-14 09:44:27.362566 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.362570 | controller |     - '        '
2026-03-14 09:44:27.362575 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.362579 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.362583 | controller |       | base64 -d)'
2026-03-14 09:44:27.362587 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.362591 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.362596 | controller |     - '          exit 1'
2026-03-14 09:44:27.362600 | controller |     - '        fi'
2026-03-14 09:44:27.362604 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.362608 | controller |     - '        '
2026-03-14 09:44:27.362614 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.362619 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.362623 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.362627 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.362631 | controller |       --timeout=300s'
2026-03-14 09:44:27.362639 | controller |     - '        '
2026-03-14 09:44:27.362645 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.362650 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.362656 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.362662 | controller |     - '        '
2026-03-14 09:44:27.362667 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.362671 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.362675 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.362679 | controller |     - '        '
2026-03-14 09:44:27.362683 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.362688 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.362692 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.362696 | controller |       available"'
2026-03-14 09:44:27.362700 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.362704 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.362708 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.362712 | controller |     - '          exit 1'
2026-03-14 09:44:27.362717 | controller |     - '        fi'
2026-03-14 09:44:27.362721 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.362725 | controller |     - '        '
2026-03-14 09:44:27.362729 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.362733 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.362737 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.362742 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.362746 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.362750 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.362754 | controller |       -- \'
2026-03-14 09:44:27.362758 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.362762 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.362767 | controller |     - '    logger.go:42: 09:33:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.362771 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.362777 | controller |     - '    logger.go:42: 09:33:44 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.362783 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.362788 | controller |       found'
2026-03-14 09:44:27.362793 | controller |     - '    logger.go:42: 09:33:45 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.362800 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.362823 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.362827 | controller |     - '        '
2026-03-14 09:44:27.362832 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.362836 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.362841 | controller |     - '        '
2026-03-14 09:44:27.362854 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.362859 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.362863 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.362867 | controller |     - '          exit 1'
2026-03-14 09:44:27.362872 | controller |     - '        fi'
2026-03-14 09:44:27.362876 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.362880 | controller |     - '        '
2026-03-14 09:44:27.362885 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.362889 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.362893 | controller |       | base64 -d)'
2026-03-14 09:44:27.362897 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.362902 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.362906 | controller |     - '          exit 1'
2026-03-14 09:44:27.362910 | controller |     - '        fi'
2026-03-14 09:44:27.362914 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.362918 | controller |     - '        '
2026-03-14 09:44:27.362923 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.362927 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.362931 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.362935 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.362940 | controller |       --timeout=300s'
2026-03-14 09:44:27.362944 | controller |     - '        '
2026-03-14 09:44:27.362948 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.362952 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.362956 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.362961 | controller |     - '        '
2026-03-14 09:44:27.362965 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.362969 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.362973 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.362977 | controller |     - '        '
2026-03-14 09:44:27.362982 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.362986 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.362990 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.362996 | controller |       available"'
2026-03-14 09:44:27.363002 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.363007 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.363013 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.363019 | controller |     - '          exit 1'
2026-03-14 09:44:27.363025 | controller |     - '        fi'
2026-03-14 09:44:27.363030 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.363040 | controller |     - '        '
2026-03-14 09:44:27.363046 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.363051 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.363055 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.363061 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.363066 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.363072 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.363078 | controller |       -- \'
2026-03-14 09:44:27.363084 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.363089 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.363099 | controller |     - '    logger.go:42: 09:33:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.363104 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.363110 | controller |     - '    logger.go:42: 09:33:45 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.363116 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.363122 | controller |       found'
2026-03-14 09:44:27.363128 | controller |     - '    logger.go:42: 09:33:46 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.363136 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.363142 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.363147 | controller |     - '        '
2026-03-14 09:44:27.363152 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.363156 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.363160 | controller |     - '        '
2026-03-14 09:44:27.363164 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.363176 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.363181 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.363186 | controller |     - '          exit 1'
2026-03-14 09:44:27.363190 | controller |     - '        fi'
2026-03-14 09:44:27.363194 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.363198 | controller |     - '        '
2026-03-14 09:44:27.363203 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.363207 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.363211 | controller |       | base64 -d)'
2026-03-14 09:44:27.363215 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.363219 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.363224 | controller |     - '          exit 1'
2026-03-14 09:44:27.363228 | controller |     - '        fi'
2026-03-14 09:44:27.363232 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.363236 | controller |     - '        '
2026-03-14 09:44:27.363241 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.363245 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.363249 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.363253 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.363257 | controller |       --timeout=300s'
2026-03-14 09:44:27.363262 | controller |     - '        '
2026-03-14 09:44:27.363266 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.363273 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.363277 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.363281 | controller |     - '        '
2026-03-14 09:44:27.363285 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.363305 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.363313 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.363317 | controller |     - '        '
2026-03-14 09:44:27.363321 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.363325 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.363330 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.363334 | controller |       available"'
2026-03-14 09:44:27.363338 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.363342 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.363346 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.363351 | controller |     - '          exit 1'
2026-03-14 09:44:27.363355 | controller |     - '        fi'
2026-03-14 09:44:27.363359 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.363363 | controller |     - '        '
2026-03-14 09:44:27.363367 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.363372 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.363376 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.363380 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.363384 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.363388 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.363392 | controller |       -- \'
2026-03-14 09:44:27.363397 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.363401 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.363405 | controller |     - '    logger.go:42: 09:33:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.363409 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.363415 | controller |     - '    logger.go:42: 09:33:47 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.363421 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.363426 | controller |       found'
2026-03-14 09:44:27.363432 | controller |     - '    logger.go:42: 09:33:48 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.363438 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.363444 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.363450 | controller |     - '        '
2026-03-14 09:44:27.363456 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.363461 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.363465 | controller |     - '        '
2026-03-14 09:44:27.363470 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.363479 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.363493 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.363499 | controller |     - '          exit 1'
2026-03-14 09:44:27.363505 | controller |     - '        fi'
2026-03-14 09:44:27.363511 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.363517 | controller |     - '        '
2026-03-14 09:44:27.363525 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.363530 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.363535 | controller |       | base64 -d)'
2026-03-14 09:44:27.363541 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.363547 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.363553 | controller |     - '          exit 1'
2026-03-14 09:44:27.363558 | controller |     - '        fi'
2026-03-14 09:44:27.363564 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.363570 | controller |     - '        '
2026-03-14 09:44:27.363576 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.363582 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.363587 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.363593 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.363599 | controller |       --timeout=300s'
2026-03-14 09:44:27.363605 | controller |     - '        '
2026-03-14 09:44:27.363610 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.363616 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.363622 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.363628 | controller |     - '        '
2026-03-14 09:44:27.363633 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.363639 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.363645 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.363651 | controller |     - '        '
2026-03-14 09:44:27.363656 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.363662 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.363668 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.363674 | controller |       available"'
2026-03-14 09:44:27.363680 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.363686 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.363691 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.363697 | controller |     - '          exit 1'
2026-03-14 09:44:27.363706 | controller |     - '        fi'
2026-03-14 09:44:27.363712 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.363718 | controller |     - '        '
2026-03-14 09:44:27.363724 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.363730 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.363736 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.363742 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.363751 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.363756 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.363762 | controller |       -- \'
2026-03-14 09:44:27.363768 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.363774 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.363784 | controller |     - '    logger.go:42: 09:33:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.363790 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.363796 | controller |     - '    logger.go:42: 09:33:48 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.363802 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.363808 | controller |       found'
2026-03-14 09:44:27.363814 | controller |     - '    logger.go:42: 09:33:49 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.363820 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.363825 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.363831 | controller |     - '        '
2026-03-14 09:44:27.363839 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.363845 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.363851 | controller |     - '        '
2026-03-14 09:44:27.363857 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.363862 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.363868 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.363882 | controller |     - '          exit 1'
2026-03-14 09:44:27.363888 | controller |     - '        fi'
2026-03-14 09:44:27.363893 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.363898 | controller |     - '        '
2026-03-14 09:44:27.363904 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.363910 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.363916 | controller |       | base64 -d)'
2026-03-14 09:44:27.363922 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.363927 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.363933 | controller |     - '          exit 1'
2026-03-14 09:44:27.363938 | controller |     - '        fi'
2026-03-14 09:44:27.363944 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.363949 | controller |     - '        '
2026-03-14 09:44:27.363955 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.363960 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.363966 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.363971 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.363977 | controller |       --timeout=300s'
2026-03-14 09:44:27.363983 | controller |     - '        '
2026-03-14 09:44:27.363988 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.363994 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.363999 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.364005 | controller |     - '        '
2026-03-14 09:44:27.364010 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.364016 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.364025 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.364031 | controller |     - '        '
2026-03-14 09:44:27.364037 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.364042 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.364048 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.364053 | controller |       available"'
2026-03-14 09:44:27.364059 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.364065 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.364071 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.364077 | controller |     - '          exit 1'
2026-03-14 09:44:27.364083 | controller |     - '        fi'
2026-03-14 09:44:27.364089 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.364094 | controller |     - '        '
2026-03-14 09:44:27.364100 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.364106 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.364112 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.364117 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.364123 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.364129 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.364135 | controller |       -- \'
2026-03-14 09:44:27.364141 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.364146 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.364152 | controller |     - '    logger.go:42: 09:33:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.364158 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.364164 | controller |     - '    logger.go:42: 09:33:49 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.364170 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.364175 | controller |       found'
2026-03-14 09:44:27.364181 | controller |     - '    logger.go:42: 09:33:50 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.364187 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.364193 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.364198 | controller |     - '        '
2026-03-14 09:44:27.364204 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.364210 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.364216 | controller |     - '        '
2026-03-14 09:44:27.364222 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.364227 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.364233 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.364237 | controller |     - '          exit 1'
2026-03-14 09:44:27.364241 | controller |     - '        fi'
2026-03-14 09:44:27.364253 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.364258 | controller |     - '        '
2026-03-14 09:44:27.364262 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.364266 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.364273 | controller |       | base64 -d)'
2026-03-14 09:44:27.364277 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.364309 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.364318 | controller |     - '          exit 1'
2026-03-14 09:44:27.364323 | controller |     - '        fi'
2026-03-14 09:44:27.364327 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.364331 | controller |     - '        '
2026-03-14 09:44:27.364335 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.364340 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.364344 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.364348 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.364352 | controller |       --timeout=300s'
2026-03-14 09:44:27.364356 | controller |     - '        '
2026-03-14 09:44:27.364361 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.364365 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.364369 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.364373 | controller |     - '        '
2026-03-14 09:44:27.364377 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.364382 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.364386 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.364390 | controller |     - '        '
2026-03-14 09:44:27.364394 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.364400 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.364405 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.364409 | controller |       available"'
2026-03-14 09:44:27.364413 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.364417 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.364422 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.364426 | controller |     - '          exit 1'
2026-03-14 09:44:27.364430 | controller |     - '        fi'
2026-03-14 09:44:27.364434 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.364438 | controller |     - '        '
2026-03-14 09:44:27.364443 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.364447 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.364451 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.364455 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.364459 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.364464 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.364468 | controller |       -- \'
2026-03-14 09:44:27.364472 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.364476 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.364482 | controller |     - '    logger.go:42: 09:33:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.364487 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.364491 | controller |     - '    logger.go:42: 09:33:50 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.364495 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.364499 | controller |       found'
2026-03-14 09:44:27.364503 | controller |     - '    logger.go:42: 09:33:51 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.364508 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.364512 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.364516 | controller |     - '        '
2026-03-14 09:44:27.364520 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.364526 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.364530 | controller |     - '        '
2026-03-14 09:44:27.364535 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.364539 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.364543 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.364547 | controller |     - '          exit 1'
2026-03-14 09:44:27.364552 | controller |     - '        fi'
2026-03-14 09:44:27.364556 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.364560 | controller |     - '        '
2026-03-14 09:44:27.364570 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.364575 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.364579 | controller |       | base64 -d)'
2026-03-14 09:44:27.364583 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.364588 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.364592 | controller |     - '          exit 1'
2026-03-14 09:44:27.364596 | controller |     - '        fi'
2026-03-14 09:44:27.364600 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.364604 | controller |     - '        '
2026-03-14 09:44:27.364609 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.364613 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.364617 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.364621 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.364625 | controller |       --timeout=300s'
2026-03-14 09:44:27.364631 | controller |     - '        '
2026-03-14 09:44:27.364637 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.364642 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.364648 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.364654 | controller |     - '        '
2026-03-14 09:44:27.364659 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.364663 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.364668 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.364672 | controller |     - '        '
2026-03-14 09:44:27.364676 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.364680 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.364685 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.364691 | controller |       available"'
2026-03-14 09:44:27.364695 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.364700 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.364704 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.364708 | controller |     - '          exit 1'
2026-03-14 09:44:27.364712 | controller |     - '        fi'
2026-03-14 09:44:27.364716 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.364721 | controller |     - '        '
2026-03-14 09:44:27.364725 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.364729 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.364733 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.364737 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.364742 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.364746 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.364750 | controller |       -- \'
2026-03-14 09:44:27.364754 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.364759 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.364763 | controller |     - '    logger.go:42: 09:33:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.364767 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.364771 | controller |     - '    logger.go:42: 09:33:51 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.364775 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.364780 | controller |       found'
2026-03-14 09:44:27.364787 | controller |     - '    logger.go:42: 09:33:52 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.364792 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.364796 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.364800 | controller |     - '        '
2026-03-14 09:44:27.364804 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.364809 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.364813 | controller |     - '        '
2026-03-14 09:44:27.364817 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.364821 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.364825 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.364830 | controller |     - '          exit 1'
2026-03-14 09:44:27.364834 | controller |     - '        fi'
2026-03-14 09:44:27.364838 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.364842 | controller |     - '        '
2026-03-14 09:44:27.364853 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.364857 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.364862 | controller |       | base64 -d)'
2026-03-14 09:44:27.364866 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.364870 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.364874 | controller |     - '          exit 1'
2026-03-14 09:44:27.364879 | controller |     - '        fi'
2026-03-14 09:44:27.364883 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.364887 | controller |     - '        '
2026-03-14 09:44:27.364891 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.364897 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.364902 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.364906 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.364910 | controller |       --timeout=300s'
2026-03-14 09:44:27.364917 | controller |     - '        '
2026-03-14 09:44:27.364922 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.364928 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.364933 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.364937 | controller |     - '        '
2026-03-14 09:44:27.364942 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.364947 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.364952 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.364956 | controller |     - '        '
2026-03-14 09:44:27.364960 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.364965 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.364969 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.364973 | controller |       available"'
2026-03-14 09:44:27.364977 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.364983 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.364987 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.364992 | controller |     - '          exit 1'
2026-03-14 09:44:27.364996 | controller |     - '        fi'
2026-03-14 09:44:27.365000 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.365004 | controller |     - '        '
2026-03-14 09:44:27.365008 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.365013 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.365017 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.365021 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.365025 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.365029 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.365034 | controller |       -- \'
2026-03-14 09:44:27.365038 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.365043 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.365047 | controller |     - '    logger.go:42: 09:33:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.365052 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.365070 | controller |     - '    logger.go:42: 09:33:52 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.365074 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.365078 | controller |       found'
2026-03-14 09:44:27.365083 | controller |     - '    logger.go:42: 09:33:53 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.365087 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.365093 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.365097 | controller |     - '        '
2026-03-14 09:44:27.365101 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.365105 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.365110 | controller |     - '        '
2026-03-14 09:44:27.365116 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.365120 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.365125 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.365129 | controller |     - '          exit 1'
2026-03-14 09:44:27.365133 | controller |     - '        fi'
2026-03-14 09:44:27.365137 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.365142 | controller |     - '        '
2026-03-14 09:44:27.365146 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.365156 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.365161 | controller |       | base64 -d)'
2026-03-14 09:44:27.365165 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.365169 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.365173 | controller |     - '          exit 1'
2026-03-14 09:44:27.365178 | controller |     - '        fi'
2026-03-14 09:44:27.365182 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.365186 | controller |     - '        '
2026-03-14 09:44:27.365190 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.365195 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.365199 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.365203 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.365207 | controller |       --timeout=300s'
2026-03-14 09:44:27.365211 | controller |     - '        '
2026-03-14 09:44:27.365216 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.365220 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.365224 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.365228 | controller |     - '        '
2026-03-14 09:44:27.365234 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.365238 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.365243 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.365247 | controller |     - '        '
2026-03-14 09:44:27.365251 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.365255 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.365259 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.365264 | controller |       available"'
2026-03-14 09:44:27.365269 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.365275 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.365281 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.365290 | controller |     - '          exit 1'
2026-03-14 09:44:27.365312 | controller |     - '        fi'
2026-03-14 09:44:27.365318 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.365326 | controller |     - '        '
2026-03-14 09:44:27.365330 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.365334 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.365339 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.365343 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.365347 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.365351 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.365355 | controller |       -- \'
2026-03-14 09:44:27.365360 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.365364 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.365368 | controller |     - '    logger.go:42: 09:33:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.365372 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.365377 | controller |     - '    logger.go:42: 09:33:54 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.365381 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.365385 | controller |       found'
2026-03-14 09:44:27.365389 | controller |     - '    logger.go:42: 09:33:55 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.365394 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.365398 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.365402 | controller |     - '        '
2026-03-14 09:44:27.365406 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.365410 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.365415 | controller |     - '        '
2026-03-14 09:44:27.365419 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.365423 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.365427 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.365432 | controller |     - '          exit 1'
2026-03-14 09:44:27.365436 | controller |     - '        fi'
2026-03-14 09:44:27.365440 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.365444 | controller |     - '        '
2026-03-14 09:44:27.365448 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.365459 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.365464 | controller |       | base64 -d)'
2026-03-14 09:44:27.365468 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.365472 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.365476 | controller |     - '          exit 1'
2026-03-14 09:44:27.365481 | controller |     - '        fi'
2026-03-14 09:44:27.365485 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.365489 | controller |     - '        '
2026-03-14 09:44:27.365493 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.365498 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.365502 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.365506 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.365510 | controller |       --timeout=300s'
2026-03-14 09:44:27.365515 | controller |     - '        '
2026-03-14 09:44:27.365519 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.365525 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.365531 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.365535 | controller |     - '        '
2026-03-14 09:44:27.365540 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.365544 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.365548 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.365552 | controller |     - '        '
2026-03-14 09:44:27.365557 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.365561 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.365565 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.365569 | controller |       available"'
2026-03-14 09:44:27.365573 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.365578 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.365582 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.365586 | controller |     - '          exit 1'
2026-03-14 09:44:27.365590 | controller |     - '        fi'
2026-03-14 09:44:27.365595 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.365599 | controller |     - '        '
2026-03-14 09:44:27.365603 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.365607 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.365611 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.365616 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.365620 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.365624 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.365628 | controller |       -- \'
2026-03-14 09:44:27.365632 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.365637 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.365641 | controller |     - '    logger.go:42: 09:33:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.365645 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.365649 | controller |     - '    logger.go:42: 09:33:55 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.365654 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.365658 | controller |       found'
2026-03-14 09:44:27.365662 | controller |     - '    logger.go:42: 09:33:56 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.365666 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.365670 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.365675 | controller |     - '        '
2026-03-14 09:44:27.365679 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.365683 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.365687 | controller |     - '        '
2026-03-14 09:44:27.365692 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.365696 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.365703 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.365760 | controller |     - '          exit 1'
2026-03-14 09:44:27.365766 | controller |     - '        fi'
2026-03-14 09:44:27.365771 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.365775 | controller |     - '        '
2026-03-14 09:44:27.365779 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.365784 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.365788 | controller |       | base64 -d)'
2026-03-14 09:44:27.365799 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.365804 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.365808 | controller |     - '          exit 1'
2026-03-14 09:44:27.365813 | controller |     - '        fi'
2026-03-14 09:44:27.365817 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.365821 | controller |     - '        '
2026-03-14 09:44:27.365826 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.365830 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.365837 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.365841 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.365846 | controller |       --timeout=300s'
2026-03-14 09:44:27.365850 | controller |     - '        '
2026-03-14 09:44:27.365854 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.365859 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.365863 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.365867 | controller |     - '        '
2026-03-14 09:44:27.365872 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.365876 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.365880 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.365885 | controller |     - '        '
2026-03-14 09:44:27.365889 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.365893 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.365897 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.365902 | controller |       available"'
2026-03-14 09:44:27.365906 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.365910 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.365915 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.365919 | controller |     - '          exit 1'
2026-03-14 09:44:27.365923 | controller |     - '        fi'
2026-03-14 09:44:27.365928 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.365932 | controller |     - '        '
2026-03-14 09:44:27.365936 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.365941 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.365945 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.365949 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.365954 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.365960 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.365964 | controller |       -- \'
2026-03-14 09:44:27.365970 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.365975 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.365980 | controller |     - '    logger.go:42: 09:33:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.365985 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.365990 | controller |     - '    logger.go:42: 09:33:56 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.365996 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.366001 | controller |       found'
2026-03-14 09:44:27.366005 | controller |     - '    logger.go:42: 09:33:57 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.366010 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.366014 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.366018 | controller |     - '        '
2026-03-14 09:44:27.366023 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.366027 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.366031 | controller |     - '        '
2026-03-14 09:44:27.366036 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.366040 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.366045 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.366067 | controller |     - '          exit 1'
2026-03-14 09:44:27.366073 | controller |     - '        fi'
2026-03-14 09:44:27.366078 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.366082 | controller |     - '        '
2026-03-14 09:44:27.366089 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.366093 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.366098 | controller |       | base64 -d)'
2026-03-14 09:44:27.366102 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.366112 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.366118 | controller |     - '          exit 1'
2026-03-14 09:44:27.366122 | controller |     - '        fi'
2026-03-14 09:44:27.366126 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.366131 | controller |     - '        '
2026-03-14 09:44:27.366135 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.366139 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.366144 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.366148 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.366152 | controller |       --timeout=300s'
2026-03-14 09:44:27.366157 | controller |     - '        '
2026-03-14 09:44:27.366161 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.366165 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.366169 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.366174 | controller |     - '        '
2026-03-14 09:44:27.366178 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.366182 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.366189 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.366193 | controller |     - '        '
2026-03-14 09:44:27.366197 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.366201 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.366206 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.366210 | controller |       available"'
2026-03-14 09:44:27.366214 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.366218 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.366223 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.366227 | controller |     - '          exit 1'
2026-03-14 09:44:27.366231 | controller |     - '        fi'
2026-03-14 09:44:27.366236 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.366240 | controller |     - '        '
2026-03-14 09:44:27.366244 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.366248 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.366253 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.366257 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.366261 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.366265 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.366270 | controller |       -- \'
2026-03-14 09:44:27.366274 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.366278 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.366282 | controller |     - '    logger.go:42: 09:33:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.366287 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.366308 | controller |     - '    logger.go:42: 09:33:57 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.366313 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.366318 | controller |       found'
2026-03-14 09:44:27.366322 | controller |     - '    logger.go:42: 09:33:58 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.366326 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.366331 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.366335 | controller |     - '        '
2026-03-14 09:44:27.366339 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.366343 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.366348 | controller |     - '        '
2026-03-14 09:44:27.366352 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.366356 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.366361 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.366365 | controller |     - '          exit 1'
2026-03-14 09:44:27.366369 | controller |     - '        fi'
2026-03-14 09:44:27.366373 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.366378 | controller |     - '        '
2026-03-14 09:44:27.366384 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.366388 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.366393 | controller |       | base64 -d)'
2026-03-14 09:44:27.366399 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.366410 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.366415 | controller |     - '          exit 1'
2026-03-14 09:44:27.366419 | controller |     - '        fi'
2026-03-14 09:44:27.366423 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.366428 | controller |     - '        '
2026-03-14 09:44:27.366432 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.366436 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.366442 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.366446 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.366451 | controller |       --timeout=300s'
2026-03-14 09:44:27.366455 | controller |     - '        '
2026-03-14 09:44:27.366459 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.366464 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.366468 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.366472 | controller |     - '        '
2026-03-14 09:44:27.366477 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.366481 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.366485 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.366489 | controller |     - '        '
2026-03-14 09:44:27.366494 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.366498 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.366502 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.366507 | controller |       available"'
2026-03-14 09:44:27.366511 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.366515 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.366520 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.366524 | controller |     - '          exit 1'
2026-03-14 09:44:27.366528 | controller |     - '        fi'
2026-03-14 09:44:27.366532 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.366537 | controller |     - '        '
2026-03-14 09:44:27.366541 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.366545 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.366550 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.366554 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.366560 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.366564 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.366569 | controller |       -- \'
2026-03-14 09:44:27.366573 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.366577 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.366581 | controller |     - '    logger.go:42: 09:33:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.366587 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.366592 | controller |     - '    logger.go:42: 09:33:58 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.366596 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.366600 | controller |       found'
2026-03-14 09:44:27.366605 | controller |     - '    logger.go:42: 09:33:59 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.366609 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.366613 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.366618 | controller |     - '        '
2026-03-14 09:44:27.366622 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.366626 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.366630 | controller |     - '        '
2026-03-14 09:44:27.366635 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.366639 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.366643 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.366647 | controller |     - '          exit 1'
2026-03-14 09:44:27.366652 | controller |     - '        fi'
2026-03-14 09:44:27.366687 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.366694 | controller |     - '        '
2026-03-14 09:44:27.366698 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.366702 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.366707 | controller |       | base64 -d)'
2026-03-14 09:44:27.366711 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.366719 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.366761 | controller |     - '          exit 1'
2026-03-14 09:44:27.366767 | controller |     - '        fi'
2026-03-14 09:44:27.366772 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.366779 | controller |     - '        '
2026-03-14 09:44:27.366786 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.366790 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.366797 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.366801 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.366806 | controller |       --timeout=300s'
2026-03-14 09:44:27.366810 | controller |     - '        '
2026-03-14 09:44:27.366815 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.366821 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.366826 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.366830 | controller |     - '        '
2026-03-14 09:44:27.366834 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.366842 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.366846 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.366851 | controller |     - '        '
2026-03-14 09:44:27.366856 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.366860 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.366867 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.366881 | controller |       available"'
2026-03-14 09:44:27.366888 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.366896 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.366901 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.366905 | controller |     - '          exit 1'
2026-03-14 09:44:27.366910 | controller |     - '        fi'
2026-03-14 09:44:27.366914 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.366923 | controller |     - '        '
2026-03-14 09:44:27.366928 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.366933 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.366938 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.366946 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.366959 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.366963 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.366967 | controller |       -- \'
2026-03-14 09:44:27.366972 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.366976 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.366980 | controller |     - '    logger.go:42: 09:33:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.366984 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.366989 | controller |     - '    logger.go:42: 09:33:59 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.366993 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.366997 | controller |       found'
2026-03-14 09:44:27.367001 | controller |     - '    logger.go:42: 09:34:00 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.367008 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.367020 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.367024 | controller |     - '        '
2026-03-14 09:44:27.367029 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.367033 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.367037 | controller |     - '        '
2026-03-14 09:44:27.367042 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.367046 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.367050 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.367055 | controller |     - '          exit 1'
2026-03-14 09:44:27.367061 | controller |     - '        fi'
2026-03-14 09:44:27.367075 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.367079 | controller |     - '        '
2026-03-14 09:44:27.367083 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.367087 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.367092 | controller |       | base64 -d)'
2026-03-14 09:44:27.367098 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.367110 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.367114 | controller |     - '          exit 1'
2026-03-14 09:44:27.367121 | controller |     - '        fi'
2026-03-14 09:44:27.367136 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.367142 | controller |     - '        '
2026-03-14 09:44:27.367146 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.367150 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.367157 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.367168 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.367172 | controller |       --timeout=300s'
2026-03-14 09:44:27.367177 | controller |     - '        '
2026-03-14 09:44:27.367182 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.367186 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.367191 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.367196 | controller |     - '        '
2026-03-14 09:44:27.367227 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.367233 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.367237 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.367241 | controller |     - '        '
2026-03-14 09:44:27.367245 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.367250 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.367254 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.367258 | controller |       available"'
2026-03-14 09:44:27.367262 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.367267 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.367271 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.367275 | controller |     - '          exit 1'
2026-03-14 09:44:27.367279 | controller |     - '        fi'
2026-03-14 09:44:27.367284 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.367304 | controller |     - '        '
2026-03-14 09:44:27.367312 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.367316 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.367320 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.367324 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.367329 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.367333 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.367337 | controller |       -- \'
2026-03-14 09:44:27.367341 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.367345 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.367350 | controller |     - '    logger.go:42: 09:34:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.367354 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.367361 | controller |     - '    logger.go:42: 09:34:01 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.367365 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.367370 | controller |       found'
2026-03-14 09:44:27.367374 | controller |     - '    logger.go:42: 09:34:02 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.367378 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.367382 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.367388 | controller |     - '        '
2026-03-14 09:44:27.367393 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.367397 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.367401 | controller |     - '        '
2026-03-14 09:44:27.367405 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.367409 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.367414 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.367418 | controller |     - '          exit 1'
2026-03-14 09:44:27.367422 | controller |     - '        fi'
2026-03-14 09:44:27.367426 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.367431 | controller |     - '        '
2026-03-14 09:44:27.367435 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.367439 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.367443 | controller |       | base64 -d)'
2026-03-14 09:44:27.367447 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.367452 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.367456 | controller |     - '          exit 1'
2026-03-14 09:44:27.367460 | controller |     - '        fi'
2026-03-14 09:44:27.367464 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.367475 | controller |     - '        '
2026-03-14 09:44:27.367480 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.367484 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.367488 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.367492 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.367496 | controller |       --timeout=300s'
2026-03-14 09:44:27.367501 | controller |     - '        '
2026-03-14 09:44:27.367505 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.367509 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.367513 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.367517 | controller |     - '        '
2026-03-14 09:44:27.367522 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.367526 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.367530 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.367534 | controller |     - '        '
2026-03-14 09:44:27.367538 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.367542 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.367547 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.367551 | controller |       available"'
2026-03-14 09:44:27.367555 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.367559 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.367563 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.367568 | controller |     - '          exit 1'
2026-03-14 09:44:27.367572 | controller |     - '        fi'
2026-03-14 09:44:27.367576 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.367580 | controller |     - '        '
2026-03-14 09:44:27.367584 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.367590 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.367594 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.367599 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.367603 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.367607 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.367611 | controller |       -- \'
2026-03-14 09:44:27.367615 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.367620 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.367624 | controller |     - '    logger.go:42: 09:34:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.367628 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.367632 | controller |     - '    logger.go:42: 09:34:02 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.367636 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.367641 | controller |       found'
2026-03-14 09:44:27.367645 | controller |     - '    logger.go:42: 09:34:03 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.367649 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.367653 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.367657 | controller |     - '        '
2026-03-14 09:44:27.367662 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.367666 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.367692 | controller |     - '        '
2026-03-14 09:44:27.367698 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.367703 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.367707 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.367711 | controller |     - '          exit 1'
2026-03-14 09:44:27.367715 | controller |     - '        fi'
2026-03-14 09:44:27.367719 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.367724 | controller |     - '        '
2026-03-14 09:44:27.367728 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.367732 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.367736 | controller |       | base64 -d)'
2026-03-14 09:44:27.367740 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.367745 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.367749 | controller |     - '          exit 1'
2026-03-14 09:44:27.367753 | controller |     - '        fi'
2026-03-14 09:44:27.367757 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.367762 | controller |     - '        '
2026-03-14 09:44:27.367772 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.367776 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.367781 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.367785 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.367789 | controller |       --timeout=300s'
2026-03-14 09:44:27.367793 | controller |     - '        '
2026-03-14 09:44:27.367797 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.367802 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.367808 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.367812 | controller |     - '        '
2026-03-14 09:44:27.367817 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.367823 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.367827 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.367831 | controller |     - '        '
2026-03-14 09:44:27.367837 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.367841 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.367845 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.367850 | controller |       available"'
2026-03-14 09:44:27.367854 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.367859 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.367864 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.367868 | controller |     - '          exit 1'
2026-03-14 09:44:27.367872 | controller |     - '        fi'
2026-03-14 09:44:27.367876 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.367880 | controller |     - '        '
2026-03-14 09:44:27.367885 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.367889 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.367893 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.367897 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.367902 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.367906 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.367910 | controller |       -- \'
2026-03-14 09:44:27.367914 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.367919 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.367924 | controller |     - '    logger.go:42: 09:34:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.367929 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.367934 | controller |     - '    logger.go:42: 09:34:03 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.367938 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.367943 | controller |       found'
2026-03-14 09:44:27.367948 | controller |     - '    logger.go:42: 09:34:04 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.367953 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.367957 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.367962 | controller |     - '        '
2026-03-14 09:44:27.367966 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.367970 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.367974 | controller |     - '        '
2026-03-14 09:44:27.367979 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.367983 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.367987 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.367993 | controller |     - '          exit 1'
2026-03-14 09:44:27.367997 | controller |     - '        fi'
2026-03-14 09:44:27.368001 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.368006 | controller |     - '        '
2026-03-14 09:44:27.368010 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.368014 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.368018 | controller |       | base64 -d)'
2026-03-14 09:44:27.368022 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.368027 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.368031 | controller |     - '          exit 1'
2026-03-14 09:44:27.368035 | controller |     - '        fi'
2026-03-14 09:44:27.368039 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.368043 | controller |     - '        '
2026-03-14 09:44:27.368048 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.368057 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.368062 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.368068 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.368074 | controller |       --timeout=300s'
2026-03-14 09:44:27.368080 | controller |     - '        '
2026-03-14 09:44:27.368086 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.368092 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.368098 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.368104 | controller |     - '        '
2026-03-14 09:44:27.368109 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.368114 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.368120 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.368126 | controller |     - '        '
2026-03-14 09:44:27.368132 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.368137 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.368141 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.368145 | controller |       available"'
2026-03-14 09:44:27.368150 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.368154 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.368158 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.368162 | controller |     - '          exit 1'
2026-03-14 09:44:27.368167 | controller |     - '        fi'
2026-03-14 09:44:27.368171 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.368175 | controller |     - '        '
2026-03-14 09:44:27.368179 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.368183 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.368188 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.368192 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.368210 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.368218 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.368222 | controller |       -- \'
2026-03-14 09:44:27.368227 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.368231 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.368238 | controller |     - '    logger.go:42: 09:34:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.368242 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.368247 | controller |     - '    logger.go:42: 09:34:04 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.368251 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.368255 | controller |       found'
2026-03-14 09:44:27.368259 | controller |     - '    logger.go:42: 09:34:05 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.368263 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.368268 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.368272 | controller |     - '        '
2026-03-14 09:44:27.368276 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.368280 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.368285 | controller |     - '        '
2026-03-14 09:44:27.368306 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.368311 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.368315 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.368319 | controller |     - '          exit 1'
2026-03-14 09:44:27.368323 | controller |     - '        fi'
2026-03-14 09:44:27.368328 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.368332 | controller |     - '        '
2026-03-14 09:44:27.368336 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.368340 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.368344 | controller |       | base64 -d)'
2026-03-14 09:44:27.368349 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.368353 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.368357 | controller |     - '          exit 1'
2026-03-14 09:44:27.368361 | controller |     - '        fi'
2026-03-14 09:44:27.368365 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.368370 | controller |     - '        '
2026-03-14 09:44:27.368374 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.368378 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.368389 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.368393 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.368398 | controller |       --timeout=300s'
2026-03-14 09:44:27.368402 | controller |     - '        '
2026-03-14 09:44:27.368406 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.368410 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.368415 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.368419 | controller |     - '        '
2026-03-14 09:44:27.368428 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.368432 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.368436 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.368442 | controller |     - '        '
2026-03-14 09:44:27.368447 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.368451 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.368457 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.368461 | controller |       available"'
2026-03-14 09:44:27.368465 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.368469 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.368474 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.368478 | controller |     - '          exit 1'
2026-03-14 09:44:27.368482 | controller |     - '        fi'
2026-03-14 09:44:27.368486 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.368490 | controller |     - '        '
2026-03-14 09:44:27.368495 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.368499 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.368503 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.368507 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.368512 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.368516 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.368520 | controller |       -- \'
2026-03-14 09:44:27.368524 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.368528 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.368533 | controller |     - '    logger.go:42: 09:34:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.368537 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.368541 | controller |     - '    logger.go:42: 09:34:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.368545 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.368549 | controller |       found'
2026-03-14 09:44:27.368554 | controller |     - '    logger.go:42: 09:34:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.368558 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.368562 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.368566 | controller |     - '        '
2026-03-14 09:44:27.368571 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.368575 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.368579 | controller |     - '        '
2026-03-14 09:44:27.368583 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.368587 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.368592 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.368596 | controller |     - '          exit 1'
2026-03-14 09:44:27.368600 | controller |     - '        fi'
2026-03-14 09:44:27.368604 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.368609 | controller |     - '        '
2026-03-14 09:44:27.368613 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.368617 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.368621 | controller |       | base64 -d)'
2026-03-14 09:44:27.368625 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.368631 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.368635 | controller |     - '          exit 1'
2026-03-14 09:44:27.368640 | controller |     - '        fi'
2026-03-14 09:44:27.368644 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.368648 | controller |     - '        '
2026-03-14 09:44:27.368652 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.368657 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.368666 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.368671 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.368675 | controller |       --timeout=300s'
2026-03-14 09:44:27.368679 | controller |     - '        '
2026-03-14 09:44:27.368683 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.368688 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.368692 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.368696 | controller |     - '        '
2026-03-14 09:44:27.368700 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.368705 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.368709 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.368713 | controller |     - '        '
2026-03-14 09:44:27.368717 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.368721 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.368726 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.368730 | controller |       available"'
2026-03-14 09:44:27.368734 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.368738 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.368742 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.368746 | controller |     - '          exit 1'
2026-03-14 09:44:27.368751 | controller |     - '        fi'
2026-03-14 09:44:27.368755 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.368759 | controller |     - '        '
2026-03-14 09:44:27.368763 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.368767 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.368772 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.368776 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.368780 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.368784 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.368823 | controller |       -- \'
2026-03-14 09:44:27.368828 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.368832 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.368837 | controller |     - '    logger.go:42: 09:34:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.368850 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.368858 | controller |     - '    logger.go:42: 09:34:06 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.368862 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.368867 | controller |       found'
2026-03-14 09:44:27.368871 | controller |     - '    logger.go:42: 09:34:07 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.368875 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.368880 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.368884 | controller |     - '        '
2026-03-14 09:44:27.368888 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.368893 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.368897 | controller |     - '        '
2026-03-14 09:44:27.368901 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.368905 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.368910 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.368914 | controller |     - '          exit 1'
2026-03-14 09:44:27.368918 | controller |     - '        fi'
2026-03-14 09:44:27.368923 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.368927 | controller |     - '        '
2026-03-14 09:44:27.368931 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.368935 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.368940 | controller |       | base64 -d)'
2026-03-14 09:44:27.368944 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.368948 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.368953 | controller |     - '          exit 1'
2026-03-14 09:44:27.368957 | controller |     - '        fi'
2026-03-14 09:44:27.368961 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.368965 | controller |     - '        '
2026-03-14 09:44:27.368970 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.368974 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.368978 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.368989 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.368993 | controller |       --timeout=300s'
2026-03-14 09:44:27.368998 | controller |     - '        '
2026-03-14 09:44:27.369002 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.369006 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.369011 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.369015 | controller |     - '        '
2026-03-14 09:44:27.369019 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.369024 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.369028 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.369032 | controller |     - '        '
2026-03-14 09:44:27.369036 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.369041 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.369045 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.369049 | controller |       available"'
2026-03-14 09:44:27.369054 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.369062 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.369066 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.369071 | controller |     - '          exit 1'
2026-03-14 09:44:27.369075 | controller |     - '        fi'
2026-03-14 09:44:27.369079 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.369083 | controller |     - '        '
2026-03-14 09:44:27.369088 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.369092 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.369096 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.369101 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.369105 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.369109 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.369114 | controller |       -- \'
2026-03-14 09:44:27.369118 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.369122 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.369126 | controller |     - '    logger.go:42: 09:34:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.369131 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.369135 | controller |     - '    logger.go:42: 09:34:07 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.369139 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.369143 | controller |       found'
2026-03-14 09:44:27.369148 | controller |     - '    logger.go:42: 09:34:08 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.369152 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.369156 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.369161 | controller |     - '        '
2026-03-14 09:44:27.369165 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.369169 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.369173 | controller |     - '        '
2026-03-14 09:44:27.369178 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.369182 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.369186 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.369191 | controller |     - '          exit 1'
2026-03-14 09:44:27.369195 | controller |     - '        fi'
2026-03-14 09:44:27.369199 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.369204 | controller |     - '        '
2026-03-14 09:44:27.369210 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.369215 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.369219 | controller |       | base64 -d)'
2026-03-14 09:44:27.369224 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.369228 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.369232 | controller |     - '          exit 1'
2026-03-14 09:44:27.369236 | controller |     - '        fi'
2026-03-14 09:44:27.369241 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.369245 | controller |     - '        '
2026-03-14 09:44:27.369249 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.369254 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.369260 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.369271 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.369275 | controller |       --timeout=300s'
2026-03-14 09:44:27.369280 | controller |     - '        '
2026-03-14 09:44:27.369284 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.369305 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.369312 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.369317 | controller |     - '        '
2026-03-14 09:44:27.369322 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.369326 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.369330 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.369335 | controller |     - '        '
2026-03-14 09:44:27.369339 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.369343 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.369348 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.369352 | controller |       available"'
2026-03-14 09:44:27.369356 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.369361 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.369365 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.369369 | controller |     - '          exit 1'
2026-03-14 09:44:27.369373 | controller |     - '        fi'
2026-03-14 09:44:27.369378 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.369382 | controller |     - '        '
2026-03-14 09:44:27.369386 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.369391 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.369397 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.369401 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.369406 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.369410 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.369414 | controller |       -- \'
2026-03-14 09:44:27.369418 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.369423 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.369427 | controller |     - '    logger.go:42: 09:34:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.369431 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.369436 | controller |     - '    logger.go:42: 09:34:09 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.369440 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.369444 | controller |       found'
2026-03-14 09:44:27.369448 | controller |     - '    logger.go:42: 09:34:10 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.369453 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.369457 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.369461 | controller |     - '        '
2026-03-14 09:44:27.369467 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.369472 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.369476 | controller |     - '        '
2026-03-14 09:44:27.369480 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.369484 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.369489 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.369493 | controller |     - '          exit 1'
2026-03-14 09:44:27.369497 | controller |     - '        fi'
2026-03-14 09:44:27.369502 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.369506 | controller |     - '        '
2026-03-14 09:44:27.369510 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.369514 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.369519 | controller |       | base64 -d)'
2026-03-14 09:44:27.369523 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.369527 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.369532 | controller |     - '          exit 1'
2026-03-14 09:44:27.369538 | controller |     - '        fi'
2026-03-14 09:44:27.369542 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.369546 | controller |     - '        '
2026-03-14 09:44:27.369551 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.369555 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.369559 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.369564 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.369568 | controller |       --timeout=300s'
2026-03-14 09:44:27.369572 | controller |     - '        '
2026-03-14 09:44:27.369585 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.369590 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.369594 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.369598 | controller |     - '        '
2026-03-14 09:44:27.369603 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.369607 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.369611 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.369616 | controller |     - '        '
2026-03-14 09:44:27.369620 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.369624 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.369629 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.369633 | controller |       available"'
2026-03-14 09:44:27.369637 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.369641 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.369646 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.369650 | controller |     - '          exit 1'
2026-03-14 09:44:27.369654 | controller |     - '        fi'
2026-03-14 09:44:27.369659 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.369663 | controller |     - '        '
2026-03-14 09:44:27.369669 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.369676 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.369680 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.369685 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.369689 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.369693 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.369698 | controller |       -- \'
2026-03-14 09:44:27.369702 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.369706 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.369711 | controller |     - '    logger.go:42: 09:34:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.369715 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.369719 | controller |     - '    logger.go:42: 09:34:10 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.369724 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.369728 | controller |       found'
2026-03-14 09:44:27.369732 | controller |     - '    logger.go:42: 09:34:11 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.369737 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.369743 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.369747 | controller |     - '        '
2026-03-14 09:44:27.369752 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.369756 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.369760 | controller |     - '        '
2026-03-14 09:44:27.369765 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.369769 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.369773 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.369778 | controller |     - '          exit 1'
2026-03-14 09:44:27.369782 | controller |     - '        fi'
2026-03-14 09:44:27.369786 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.369791 | controller |     - '        '
2026-03-14 09:44:27.369795 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.369799 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.369804 | controller |       | base64 -d)'
2026-03-14 09:44:27.369808 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.369812 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.369816 | controller |     - '          exit 1'
2026-03-14 09:44:27.369821 | controller |     - '        fi'
2026-03-14 09:44:27.369825 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.369829 | controller |     - '        '
2026-03-14 09:44:27.369834 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.369838 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.369842 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.369846 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.369850 | controller |       --timeout=300s'
2026-03-14 09:44:27.369855 | controller |     - '        '
2026-03-14 09:44:27.369864 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.369869 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.369876 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.369880 | controller |     - '        '
2026-03-14 09:44:27.369884 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.369889 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.369903 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.369908 | controller |     - '        '
2026-03-14 09:44:27.369913 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.369917 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.369921 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.369925 | controller |       available"'
2026-03-14 09:44:27.369930 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.369934 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.369938 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.369942 | controller |     - '          exit 1'
2026-03-14 09:44:27.369947 | controller |     - '        fi'
2026-03-14 09:44:27.369951 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.369955 | controller |     - '        '
2026-03-14 09:44:27.369960 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.369964 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.369968 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.369972 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.369977 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.369981 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.369985 | controller |       -- \'
2026-03-14 09:44:27.369990 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.369994 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.369998 | controller |     - '    logger.go:42: 09:34:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.370002 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.370007 | controller |     - '    logger.go:42: 09:34:11 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.370011 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.370015 | controller |       found'
2026-03-14 09:44:27.370020 | controller |     - '    logger.go:42: 09:34:12 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.370024 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.370028 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.370032 | controller |     - '        '
2026-03-14 09:44:27.370037 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.370041 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.370045 | controller |     - '        '
2026-03-14 09:44:27.370050 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.370057 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.370074 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.370079 | controller |     - '          exit 1'
2026-03-14 09:44:27.370087 | controller |     - '        fi'
2026-03-14 09:44:27.370092 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.370097 | controller |     - '        '
2026-03-14 09:44:27.370102 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.370107 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.370112 | controller |       | base64 -d)'
2026-03-14 09:44:27.370117 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.370121 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.370126 | controller |     - '          exit 1'
2026-03-14 09:44:27.370131 | controller |     - '        fi'
2026-03-14 09:44:27.370136 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.370141 | controller |     - '        '
2026-03-14 09:44:27.370146 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.370151 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.370156 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.370161 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.370165 | controller |       --timeout=300s'
2026-03-14 09:44:27.370170 | controller |     - '        '
2026-03-14 09:44:27.370175 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.370186 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.370191 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.370196 | controller |     - '        '
2026-03-14 09:44:27.370201 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.370206 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.370211 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.370216 | controller |     - '        '
2026-03-14 09:44:27.370221 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.370225 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.370230 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.370235 | controller |       available"'
2026-03-14 09:44:27.370240 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.370245 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.370250 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.370255 | controller |     - '          exit 1'
2026-03-14 09:44:27.370260 | controller |     - '        fi'
2026-03-14 09:44:27.370265 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.370270 | controller |     - '        '
2026-03-14 09:44:27.370274 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.370279 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.370284 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.370302 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.370310 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.370314 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.370322 | controller |       -- \'
2026-03-14 09:44:27.370327 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.370331 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.370335 | controller |     - '    logger.go:42: 09:34:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.370339 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.370344 | controller |     - '    logger.go:42: 09:34:12 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.370348 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.370352 | controller |       found'
2026-03-14 09:44:27.370356 | controller |     - '    logger.go:42: 09:34:13 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.370360 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.370365 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.370369 | controller |     - '        '
2026-03-14 09:44:27.370373 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.370377 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.370382 | controller |     - '        '
2026-03-14 09:44:27.370386 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.370390 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.370394 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.370398 | controller |     - '          exit 1'
2026-03-14 09:44:27.370403 | controller |     - '        fi'
2026-03-14 09:44:27.370407 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.370411 | controller |     - '        '
2026-03-14 09:44:27.370415 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.370420 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.370424 | controller |       | base64 -d)'
2026-03-14 09:44:27.370428 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.370432 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.370436 | controller |     - '          exit 1'
2026-03-14 09:44:27.370441 | controller |     - '        fi'
2026-03-14 09:44:27.370445 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.370449 | controller |     - '        '
2026-03-14 09:44:27.370453 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.370457 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.370462 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.370466 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.370470 | controller |       --timeout=300s'
2026-03-14 09:44:27.370474 | controller |     - '        '
2026-03-14 09:44:27.370478 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.370489 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.370494 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.370498 | controller |     - '        '
2026-03-14 09:44:27.370502 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.370506 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.370511 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.370517 | controller |     - '        '
2026-03-14 09:44:27.370521 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.370525 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.370529 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.370534 | controller |       available"'
2026-03-14 09:44:27.370538 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.370544 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.370548 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.370552 | controller |     - '          exit 1'
2026-03-14 09:44:27.370557 | controller |     - '        fi'
2026-03-14 09:44:27.370561 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.370565 | controller |     - '        '
2026-03-14 09:44:27.370569 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.370573 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.370578 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.370582 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.370586 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.370590 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.370594 | controller |       -- \'
2026-03-14 09:44:27.370599 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.370603 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.370607 | controller |     - '    logger.go:42: 09:34:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.370611 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.370615 | controller |     - '    logger.go:42: 09:34:13 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.370620 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.370624 | controller |       found'
2026-03-14 09:44:27.370628 | controller |     - '    logger.go:42: 09:34:14 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.370632 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.370637 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.370641 | controller |     - '        '
2026-03-14 09:44:27.370645 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.370649 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.370653 | controller |     - '        '
2026-03-14 09:44:27.370659 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.370665 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.370671 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.370677 | controller |     - '          exit 1'
2026-03-14 09:44:27.370683 | controller |     - '        fi'
2026-03-14 09:44:27.370689 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.370695 | controller |     - '        '
2026-03-14 09:44:27.370701 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.370706 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.370712 | controller |       | base64 -d)'
2026-03-14 09:44:27.370717 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.370723 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.370732 | controller |     - '          exit 1'
2026-03-14 09:44:27.370738 | controller |     - '        fi'
2026-03-14 09:44:27.370744 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.370749 | controller |     - '        '
2026-03-14 09:44:27.370754 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.370758 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.370762 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.370766 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.370771 | controller |       --timeout=300s'
2026-03-14 09:44:27.370775 | controller |     - '        '
2026-03-14 09:44:27.370779 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.370791 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.370795 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.370799 | controller |     - '        '
2026-03-14 09:44:27.370804 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.370808 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.370812 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.370816 | controller |     - '        '
2026-03-14 09:44:27.370821 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.370825 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.370829 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.370833 | controller |       available"'
2026-03-14 09:44:27.370837 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.370842 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.370846 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.370850 | controller |     - '          exit 1'
2026-03-14 09:44:27.370854 | controller |     - '        fi'
2026-03-14 09:44:27.370858 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.370862 | controller |     - '        '
2026-03-14 09:44:27.370867 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.370871 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.370875 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.370879 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.370883 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.370888 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.370892 | controller |       -- \'
2026-03-14 09:44:27.370896 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.370903 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.370909 | controller |     - '    logger.go:42: 09:34:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.370913 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.370918 | controller |     - '    logger.go:42: 09:34:14 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.370924 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.370928 | controller |       found'
2026-03-14 09:44:27.370932 | controller |     - '    logger.go:42: 09:34:15 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.370936 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.370941 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.370945 | controller |     - '        '
2026-03-14 09:44:27.370949 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.370953 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.370957 | controller |     - '        '
2026-03-14 09:44:27.370962 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.370966 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.370970 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.370974 | controller |     - '          exit 1'
2026-03-14 09:44:27.370978 | controller |     - '        fi'
2026-03-14 09:44:27.370983 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.370987 | controller |     - '        '
2026-03-14 09:44:27.370991 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.370995 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.371000 | controller |       | base64 -d)'
2026-03-14 09:44:27.371004 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.371008 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.371014 | controller |     - '          exit 1'
2026-03-14 09:44:27.371018 | controller |     - '        fi'
2026-03-14 09:44:27.371022 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.371027 | controller |     - '        '
2026-03-14 09:44:27.371031 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.371035 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.371039 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.371044 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.371048 | controller |       --timeout=300s'
2026-03-14 09:44:27.371052 | controller |     - '        '
2026-03-14 09:44:27.371056 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.371060 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.371071 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.371075 | controller |     - '        '
2026-03-14 09:44:27.371079 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.371084 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.371088 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.371092 | controller |     - '        '
2026-03-14 09:44:27.371096 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.371100 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.371104 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.371109 | controller |       available"'
2026-03-14 09:44:27.371113 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.371117 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.371124 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.371129 | controller |     - '          exit 1'
2026-03-14 09:44:27.371133 | controller |     - '        fi'
2026-03-14 09:44:27.371137 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.371141 | controller |     - '        '
2026-03-14 09:44:27.371145 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.371149 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.371154 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.371158 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.371162 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.371166 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.371170 | controller |       -- \'
2026-03-14 09:44:27.371175 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.371179 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.371183 | controller |     - '    logger.go:42: 09:34:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.371187 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.371191 | controller |     - '    logger.go:42: 09:34:15 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.371196 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.371200 | controller |       found'
2026-03-14 09:44:27.371204 | controller |     - '    logger.go:42: 09:34:16 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.371208 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.371212 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.371217 | controller |     - '        '
2026-03-14 09:44:27.371221 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.371225 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.371229 | controller |     - '        '
2026-03-14 09:44:27.371234 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.371238 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.371242 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.371246 | controller |     - '          exit 1'
2026-03-14 09:44:27.371250 | controller |     - '        fi'
2026-03-14 09:44:27.371254 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.371259 | controller |     - '        '
2026-03-14 09:44:27.371263 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.371268 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.371274 | controller |       | base64 -d)'
2026-03-14 09:44:27.371279 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.371285 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.371310 | controller |     - '          exit 1'
2026-03-14 09:44:27.371317 | controller |     - '        fi'
2026-03-14 09:44:27.371322 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.371328 | controller |     - '        '
2026-03-14 09:44:27.371333 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.371339 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.371345 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.371351 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.371356 | controller |       --timeout=300s'
2026-03-14 09:44:27.371362 | controller |     - '        '
2026-03-14 09:44:27.371366 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.371371 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.371383 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.371389 | controller |     - '        '
2026-03-14 09:44:27.371395 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.371400 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.371406 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.371412 | controller |     - '        '
2026-03-14 09:44:27.371417 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.371422 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.371426 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.371430 | controller |       available"'
2026-03-14 09:44:27.371434 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.371438 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.371443 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.371447 | controller |     - '          exit 1'
2026-03-14 09:44:27.371451 | controller |     - '        fi'
2026-03-14 09:44:27.371455 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.371459 | controller |     - '        '
2026-03-14 09:44:27.371464 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.371468 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.371488 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.371493 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.371497 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.371501 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.371505 | controller |       -- \'
2026-03-14 09:44:27.371510 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.371514 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.371525 | controller |     - '    logger.go:42: 09:34:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.371529 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.371534 | controller |     - '    logger.go:42: 09:34:17 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.371538 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.371542 | controller |       found'
2026-03-14 09:44:27.371546 | controller |     - '    logger.go:42: 09:34:18 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.371551 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.371558 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.371562 | controller |     - '        '
2026-03-14 09:44:27.371566 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.371572 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.371576 | controller |     - '        '
2026-03-14 09:44:27.371580 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.371585 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.371589 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.371593 | controller |     - '          exit 1'
2026-03-14 09:44:27.371597 | controller |     - '        fi'
2026-03-14 09:44:27.371602 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.371607 | controller |     - '        '
2026-03-14 09:44:27.371612 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.371629 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.371637 | controller |       | base64 -d)'
2026-03-14 09:44:27.371643 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.371649 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.371655 | controller |     - '          exit 1'
2026-03-14 09:44:27.371661 | controller |     - '        fi'
2026-03-14 09:44:27.371667 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.371672 | controller |     - '        '
2026-03-14 09:44:27.371678 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.371684 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.371689 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.371694 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.371700 | controller |       --timeout=300s'
2026-03-14 09:44:27.371705 | controller |     - '        '
2026-03-14 09:44:27.371711 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.371717 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.371723 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.371729 | controller |     - '        '
2026-03-14 09:44:27.371746 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.371754 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.371760 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.371766 | controller |     - '        '
2026-03-14 09:44:27.371770 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.371774 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.371779 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.371783 | controller |       available"'
2026-03-14 09:44:27.371792 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.371796 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.371800 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.371805 | controller |     - '          exit 1'
2026-03-14 09:44:27.371809 | controller |     - '        fi'
2026-03-14 09:44:27.371813 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.371817 | controller |     - '        '
2026-03-14 09:44:27.371822 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.371826 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.371833 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.371837 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.371841 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.371845 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.371850 | controller |       -- \'
2026-03-14 09:44:27.371854 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.371858 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.371862 | controller |     - '    logger.go:42: 09:34:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.371866 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.371870 | controller |     - '    logger.go:42: 09:34:18 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.371875 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.371879 | controller |       found'
2026-03-14 09:44:27.371883 | controller |     - '    logger.go:42: 09:34:19 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.371887 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.371891 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.371896 | controller |     - '        '
2026-03-14 09:44:27.371901 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.371921 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.371926 | controller |     - '        '
2026-03-14 09:44:27.371930 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.371934 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.371938 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.371943 | controller |     - '          exit 1'
2026-03-14 09:44:27.371947 | controller |     - '        fi'
2026-03-14 09:44:27.371951 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.371955 | controller |     - '        '
2026-03-14 09:44:27.371961 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.371965 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.371970 | controller |       | base64 -d)'
2026-03-14 09:44:27.371974 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.371978 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.371982 | controller |     - '          exit 1'
2026-03-14 09:44:27.371986 | controller |     - '        fi'
2026-03-14 09:44:27.371991 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.371997 | controller |     - '        '
2026-03-14 09:44:27.372004 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.372010 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.372016 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.372022 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.372028 | controller |       --timeout=300s'
2026-03-14 09:44:27.372033 | controller |     - '        '
2026-03-14 09:44:27.372037 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.372041 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.372045 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.372052 | controller |     - '        '
2026-03-14 09:44:27.372063 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.372068 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.372072 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.372076 | controller |     - '        '
2026-03-14 09:44:27.372081 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.372090 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.372096 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.372102 | controller |       available"'
2026-03-14 09:44:27.372108 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.372114 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.372120 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.372126 | controller |     - '          exit 1'
2026-03-14 09:44:27.372131 | controller |     - '        fi'
2026-03-14 09:44:27.372136 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.372141 | controller |     - '        '
2026-03-14 09:44:27.372145 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.372149 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.372153 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.372158 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.372162 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.372166 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.372170 | controller |       -- \'
2026-03-14 09:44:27.372174 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.372179 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.372183 | controller |     - '    logger.go:42: 09:34:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.372187 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.372191 | controller |     - '    logger.go:42: 09:34:19 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.372199 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.372203 | controller |       found'
2026-03-14 09:44:27.372207 | controller |     - '    logger.go:42: 09:34:20 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.372211 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.372215 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.372220 | controller |     - '        '
2026-03-14 09:44:27.372224 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.372228 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.372232 | controller |     - '        '
2026-03-14 09:44:27.372236 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.372241 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.372245 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.372249 | controller |     - '          exit 1'
2026-03-14 09:44:27.372253 | controller |     - '        fi'
2026-03-14 09:44:27.372259 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.372264 | controller |     - '        '
2026-03-14 09:44:27.372268 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.372272 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.372276 | controller |       | base64 -d)'
2026-03-14 09:44:27.372281 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.372285 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.372308 | controller |     - '          exit 1'
2026-03-14 09:44:27.372312 | controller |     - '        fi'
2026-03-14 09:44:27.372316 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.372320 | controller |     - '        '
2026-03-14 09:44:27.372325 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.372329 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.372333 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.372337 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.372341 | controller |       --timeout=300s'
2026-03-14 09:44:27.372346 | controller |     - '        '
2026-03-14 09:44:27.372350 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.372354 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.372358 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.372362 | controller |     - '        '
2026-03-14 09:44:27.372367 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.372378 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.372382 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.372386 | controller |     - '        '
2026-03-14 09:44:27.372391 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.372395 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.372399 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.372403 | controller |       available"'
2026-03-14 09:44:27.372407 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.372412 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.372416 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.372420 | controller |     - '          exit 1'
2026-03-14 09:44:27.372424 | controller |     - '        fi'
2026-03-14 09:44:27.372428 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.372433 | controller |     - '        '
2026-03-14 09:44:27.372437 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.372441 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.372445 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.372449 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.372454 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.372458 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.372464 | controller |       -- \'
2026-03-14 09:44:27.372468 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.372472 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.372478 | controller |     - '    logger.go:42: 09:34:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.372483 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.372487 | controller |     - '    logger.go:42: 09:34:20 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.372491 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.372495 | controller |       found'
2026-03-14 09:44:27.372500 | controller |     - '    logger.go:42: 09:34:21 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.372504 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.372508 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.372512 | controller |     - '        '
2026-03-14 09:44:27.372516 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.372521 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.372525 | controller |     - '        '
2026-03-14 09:44:27.372529 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.372533 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.372537 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.372542 | controller |     - '          exit 1'
2026-03-14 09:44:27.372546 | controller |     - '        fi'
2026-03-14 09:44:27.372550 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.372554 | controller |     - '        '
2026-03-14 09:44:27.372558 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.372563 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.372567 | controller |       | base64 -d)'
2026-03-14 09:44:27.372571 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.372575 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.372579 | controller |     - '          exit 1'
2026-03-14 09:44:27.372584 | controller |     - '        fi'
2026-03-14 09:44:27.372588 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.372592 | controller |     - '        '
2026-03-14 09:44:27.372596 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.372600 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.372605 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.372609 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.372613 | controller |       --timeout=300s'
2026-03-14 09:44:27.372618 | controller |     - '        '
2026-03-14 09:44:27.372623 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.372634 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.372646 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.372652 | controller |     - '        '
2026-03-14 09:44:27.372658 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.372674 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.372680 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.372685 | controller |     - '        '
2026-03-14 09:44:27.372696 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.372702 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.372707 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.372712 | controller |       available"'
2026-03-14 09:44:27.372718 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.372723 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.372728 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.372733 | controller |     - '          exit 1'
2026-03-14 09:44:27.372738 | controller |     - '        fi'
2026-03-14 09:44:27.372744 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.372749 | controller |     - '        '
2026-03-14 09:44:27.372755 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.372760 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.372766 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.372772 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.372778 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.372782 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.372787 | controller |       -- \'
2026-03-14 09:44:27.372791 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.372795 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.372799 | controller |     - '    logger.go:42: 09:34:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.372804 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.372808 | controller |     - '    logger.go:42: 09:34:21 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.372812 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.372816 | controller |       found'
2026-03-14 09:44:27.372824 | controller |     - '    logger.go:42: 09:34:22 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.372828 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.372833 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.372837 | controller |     - '        '
2026-03-14 09:44:27.372841 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.372845 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.372849 | controller |     - '        '
2026-03-14 09:44:27.372853 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.372857 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.372862 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.372866 | controller |     - '          exit 1'
2026-03-14 09:44:27.372870 | controller |     - '        fi'
2026-03-14 09:44:27.372874 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.372878 | controller |     - '        '
2026-03-14 09:44:27.372883 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.372887 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.372891 | controller |       | base64 -d)'
2026-03-14 09:44:27.372895 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.372899 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.372906 | controller |     - '          exit 1'
2026-03-14 09:44:27.372910 | controller |     - '        fi'
2026-03-14 09:44:27.372914 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.372919 | controller |     - '        '
2026-03-14 09:44:27.372926 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.372931 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.372935 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.372941 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.372946 | controller |       --timeout=300s'
2026-03-14 09:44:27.372950 | controller |     - '        '
2026-03-14 09:44:27.372954 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.372959 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.372963 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.372967 | controller |     - '        '
2026-03-14 09:44:27.372971 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.372975 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.372986 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.372991 | controller |     - '        '
2026-03-14 09:44:27.372995 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.372999 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.373003 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.373008 | controller |       available"'
2026-03-14 09:44:27.373012 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.373016 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.373020 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.373024 | controller |     - '          exit 1'
2026-03-14 09:44:27.373029 | controller |     - '        fi'
2026-03-14 09:44:27.373033 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.373037 | controller |     - '        '
2026-03-14 09:44:27.373041 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.373045 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.373049 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.373054 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.373058 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.373062 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.373066 | controller |       -- \'
2026-03-14 09:44:27.373070 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.373074 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.373079 | controller |     - '    logger.go:42: 09:34:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.373083 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.373087 | controller |     - '    logger.go:42: 09:34:22 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.373093 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.373097 | controller |       found'
2026-03-14 09:44:27.373101 | controller |     - '    logger.go:42: 09:34:23 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.373106 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.373110 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.373114 | controller |     - '        '
2026-03-14 09:44:27.373118 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.373122 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.373127 | controller |     - '        '
2026-03-14 09:44:27.373131 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.373135 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.373156 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.373160 | controller |     - '          exit 1'
2026-03-14 09:44:27.373165 | controller |     - '        fi'
2026-03-14 09:44:27.373169 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.373173 | controller |     - '        '
2026-03-14 09:44:27.373177 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.373181 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.373186 | controller |       | base64 -d)'
2026-03-14 09:44:27.373190 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.373194 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.373198 | controller |     - '          exit 1'
2026-03-14 09:44:27.373202 | controller |     - '        fi'
2026-03-14 09:44:27.373207 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.373211 | controller |     - '        '
2026-03-14 09:44:27.373215 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.373221 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.373235 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.373245 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.373251 | controller |       --timeout=300s'
2026-03-14 09:44:27.373257 | controller |     - '        '
2026-03-14 09:44:27.373263 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.373269 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.373277 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.373281 | controller |     - '        '
2026-03-14 09:44:27.373286 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.373308 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.373320 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.373327 | controller |     - '        '
2026-03-14 09:44:27.373333 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.373339 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.373345 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.373351 | controller |       available"'
2026-03-14 09:44:27.373357 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.373366 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.373375 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.373379 | controller |     - '          exit 1'
2026-03-14 09:44:27.373383 | controller |     - '        fi'
2026-03-14 09:44:27.373388 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.373392 | controller |     - '        '
2026-03-14 09:44:27.373396 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.373400 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.373404 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.373409 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.373413 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.373417 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.373421 | controller |       -- \'
2026-03-14 09:44:27.373425 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.373429 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.373434 | controller |     - '    logger.go:42: 09:34:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.373438 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.373442 | controller |     - '    logger.go:42: 09:34:23 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.373446 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.373451 | controller |       found'
2026-03-14 09:44:27.373455 | controller |     - '    logger.go:42: 09:34:24 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.373459 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.373463 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.373467 | controller |     - '        '
2026-03-14 09:44:27.373473 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.373478 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.373482 | controller |     - '        '
2026-03-14 09:44:27.373486 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.373490 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.373496 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.373500 | controller |     - '          exit 1'
2026-03-14 09:44:27.373504 | controller |     - '        fi'
2026-03-14 09:44:27.373509 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.373513 | controller |     - '        '
2026-03-14 09:44:27.373517 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.373521 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.373526 | controller |       | base64 -d)'
2026-03-14 09:44:27.373530 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.373534 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.373538 | controller |     - '          exit 1'
2026-03-14 09:44:27.373542 | controller |     - '        fi'
2026-03-14 09:44:27.373547 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.373551 | controller |     - '        '
2026-03-14 09:44:27.373555 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.373559 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.373563 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.373569 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.373574 | controller |       --timeout=300s'
2026-03-14 09:44:27.373578 | controller |     - '        '
2026-03-14 09:44:27.373582 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.373586 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.373591 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.373595 | controller |     - '        '
2026-03-14 09:44:27.373599 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.373603 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.373615 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.373620 | controller |     - '        '
2026-03-14 09:44:27.373624 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.373628 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.373632 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.373636 | controller |       available"'
2026-03-14 09:44:27.373641 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.373645 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.373649 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.373653 | controller |     - '          exit 1'
2026-03-14 09:44:27.373658 | controller |     - '        fi'
2026-03-14 09:44:27.373662 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.373666 | controller |     - '        '
2026-03-14 09:44:27.373670 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.373674 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.373679 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.373683 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.373687 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.373691 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.373695 | controller |       -- \'
2026-03-14 09:44:27.373700 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.373704 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.373708 | controller |     - '    logger.go:42: 09:34:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.373712 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.373716 | controller |     - '    logger.go:42: 09:34:24 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.373721 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.373725 | controller |       found'
2026-03-14 09:44:27.373729 | controller |     - '    logger.go:42: 09:34:25 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.373734 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.373738 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.373742 | controller |     - '        '
2026-03-14 09:44:27.373746 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.373750 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.373756 | controller |     - '        '
2026-03-14 09:44:27.373761 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.373765 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.373769 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.373773 | controller |     - '          exit 1'
2026-03-14 09:44:27.373778 | controller |     - '        fi'
2026-03-14 09:44:27.373782 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.373786 | controller |     - '        '
2026-03-14 09:44:27.373790 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.373794 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.373798 | controller |       | base64 -d)'
2026-03-14 09:44:27.373803 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.373807 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.373811 | controller |     - '          exit 1'
2026-03-14 09:44:27.373815 | controller |     - '        fi'
2026-03-14 09:44:27.373819 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.373823 | controller |     - '        '
2026-03-14 09:44:27.373828 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.373832 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.373836 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.373840 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.373844 | controller |       --timeout=300s'
2026-03-14 09:44:27.373848 | controller |     - '        '
2026-03-14 09:44:27.373853 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.373857 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.373861 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.373865 | controller |     - '        '
2026-03-14 09:44:27.373871 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.373875 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.373880 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.373884 | controller |     - '        '
2026-03-14 09:44:27.373893 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.373898 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.373902 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.373906 | controller |       available"'
2026-03-14 09:44:27.373910 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.373915 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.373919 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.373923 | controller |     - '          exit 1'
2026-03-14 09:44:27.373927 | controller |     - '        fi'
2026-03-14 09:44:27.373931 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.373935 | controller |     - '        '
2026-03-14 09:44:27.373940 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.373944 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.373948 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.373954 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.373958 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.373962 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.373967 | controller |       -- \'
2026-03-14 09:44:27.373971 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.373975 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.373979 | controller |     - '    logger.go:42: 09:34:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.373983 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.373988 | controller |     - '    logger.go:42: 09:34:26 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.373992 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.373996 | controller |       found'
2026-03-14 09:44:27.374000 | controller |     - '    logger.go:42: 09:34:27 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.374004 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.374009 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.374013 | controller |     - '        '
2026-03-14 09:44:27.374017 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.374023 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.374027 | controller |     - '        '
2026-03-14 09:44:27.374031 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.374035 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.374040 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.374044 | controller |     - '          exit 1'
2026-03-14 09:44:27.374048 | controller |     - '        fi'
2026-03-14 09:44:27.374055 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.374060 | controller |     - '        '
2026-03-14 09:44:27.374074 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.374080 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.374085 | controller |       | base64 -d)'
2026-03-14 09:44:27.374089 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.374093 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.374098 | controller |     - '          exit 1'
2026-03-14 09:44:27.374102 | controller |     - '        fi'
2026-03-14 09:44:27.374106 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.374110 | controller |     - '        '
2026-03-14 09:44:27.374115 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.374119 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.374123 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.374127 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.374131 | controller |       --timeout=300s'
2026-03-14 09:44:27.374135 | controller |     - '        '
2026-03-14 09:44:27.374140 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.374144 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.374148 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.374154 | controller |     - '        '
2026-03-14 09:44:27.374158 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.374163 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.374167 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.374171 | controller |     - '        '
2026-03-14 09:44:27.374181 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.374186 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.374190 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.374194 | controller |       available"'
2026-03-14 09:44:27.374199 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.374203 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.374207 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.374211 | controller |     - '          exit 1'
2026-03-14 09:44:27.374215 | controller |     - '        fi'
2026-03-14 09:44:27.374220 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.374224 | controller |     - '        '
2026-03-14 09:44:27.374228 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.374232 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.374236 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.374241 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.374247 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.374251 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.374255 | controller |       -- \'
2026-03-14 09:44:27.374260 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.374264 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.374268 | controller |     - '    logger.go:42: 09:34:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.374272 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.374276 | controller |     - '    logger.go:42: 09:34:27 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.374280 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.374285 | controller |       found'
2026-03-14 09:44:27.374305 | controller |     - '    logger.go:42: 09:34:28 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.374312 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.374317 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.374321 | controller |     - '        '
2026-03-14 09:44:27.374325 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.374329 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.374333 | controller |     - '        '
2026-03-14 09:44:27.374338 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.374342 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.374346 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.374350 | controller |     - '          exit 1'
2026-03-14 09:44:27.374354 | controller |     - '        fi'
2026-03-14 09:44:27.374359 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.374365 | controller |     - '        '
2026-03-14 09:44:27.374369 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.374373 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.374377 | controller |       | base64 -d)'
2026-03-14 09:44:27.374382 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.374386 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.374390 | controller |     - '          exit 1'
2026-03-14 09:44:27.374396 | controller |     - '        fi'
2026-03-14 09:44:27.374401 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.374405 | controller |     - '        '
2026-03-14 09:44:27.374409 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.374413 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.374417 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.374421 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.374426 | controller |       --timeout=300s'
2026-03-14 09:44:27.374432 | controller |     - '        '
2026-03-14 09:44:27.374437 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.374443 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.374449 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.374455 | controller |     - '        '
2026-03-14 09:44:27.374461 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.374467 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.374472 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.374478 | controller |     - '        '
2026-03-14 09:44:27.374485 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.374497 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.374502 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.374506 | controller |       available"'
2026-03-14 09:44:27.374511 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.374515 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.374519 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.374523 | controller |     - '          exit 1'
2026-03-14 09:44:27.374528 | controller |     - '        fi'
2026-03-14 09:44:27.374532 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.374536 | controller |     - '        '
2026-03-14 09:44:27.374540 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.374545 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.374549 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.374553 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.374557 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.374561 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.374566 | controller |       -- \'
2026-03-14 09:44:27.374570 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.374577 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.374581 | controller |     - '    logger.go:42: 09:34:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.374585 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.374589 | controller |     - '    logger.go:42: 09:34:28 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.374594 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.374598 | controller |       found'
2026-03-14 09:44:27.374602 | controller |     - '    logger.go:42: 09:34:29 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.374606 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.374610 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.374615 | controller |     - '        '
2026-03-14 09:44:27.374619 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.374623 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.374628 | controller |     - '        '
2026-03-14 09:44:27.374639 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.374655 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.374661 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.374666 | controller |     - '          exit 1'
2026-03-14 09:44:27.374670 | controller |     - '        fi'
2026-03-14 09:44:27.374674 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.374679 | controller |     - '        '
2026-03-14 09:44:27.374685 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.374690 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.374694 | controller |       | base64 -d)'
2026-03-14 09:44:27.374698 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.374702 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.374707 | controller |     - '          exit 1'
2026-03-14 09:44:27.374711 | controller |     - '        fi'
2026-03-14 09:44:27.374715 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.374719 | controller |     - '        '
2026-03-14 09:44:27.374723 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.374727 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.374732 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.374736 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.374740 | controller |       --timeout=300s'
2026-03-14 09:44:27.374744 | controller |     - '        '
2026-03-14 09:44:27.374748 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.374752 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.374756 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.374760 | controller |     - '        '
2026-03-14 09:44:27.374765 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.374769 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.374773 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.374777 | controller |     - '        '
2026-03-14 09:44:27.374781 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.374788 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.374801 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.374806 | controller |       available"'
2026-03-14 09:44:27.374810 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.374815 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.374819 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.374823 | controller |     - '          exit 1'
2026-03-14 09:44:27.374827 | controller |     - '        fi'
2026-03-14 09:44:27.374831 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.374835 | controller |     - '        '
2026-03-14 09:44:27.374840 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.374844 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.374848 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.374852 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.374856 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.374861 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.374865 | controller |       -- \'
2026-03-14 09:44:27.374869 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.374873 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.374877 | controller |     - '    logger.go:42: 09:34:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.374881 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.374886 | controller |     - '    logger.go:42: 09:34:29 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.374890 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.374894 | controller |       found'
2026-03-14 09:44:27.374898 | controller |     - '    logger.go:42: 09:34:30 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.374902 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.374907 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.374911 | controller |     - '        '
2026-03-14 09:44:27.374915 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.374919 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.374923 | controller |     - '        '
2026-03-14 09:44:27.374928 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.374932 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.374936 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.374940 | controller |     - '          exit 1'
2026-03-14 09:44:27.374944 | controller |     - '        fi'
2026-03-14 09:44:27.374948 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.374953 | controller |     - '        '
2026-03-14 09:44:27.374957 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.374962 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.374968 | controller |       | base64 -d)'
2026-03-14 09:44:27.374974 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.374980 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.374986 | controller |     - '          exit 1'
2026-03-14 09:44:27.375004 | controller |     - '        fi'
2026-03-14 09:44:27.375010 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.375016 | controller |     - '        '
2026-03-14 09:44:27.375021 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.375026 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.375032 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.375037 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.375042 | controller |       --timeout=300s'
2026-03-14 09:44:27.375048 | controller |     - '        '
2026-03-14 09:44:27.375053 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.375059 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.375064 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.375069 | controller |     - '        '
2026-03-14 09:44:27.375075 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.375080 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.375085 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.375091 | controller |     - '        '
2026-03-14 09:44:27.375096 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.375102 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.375115 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.375122 | controller |       available"'
2026-03-14 09:44:27.375127 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.375133 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.375139 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.375145 | controller |     - '          exit 1'
2026-03-14 09:44:27.375150 | controller |     - '        fi'
2026-03-14 09:44:27.375160 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.375166 | controller |     - '        '
2026-03-14 09:44:27.375172 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.375177 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.375183 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.375189 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.375194 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.375200 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.375206 | controller |       -- \'
2026-03-14 09:44:27.375211 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.375216 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.375222 | controller |     - '    logger.go:42: 09:34:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.375228 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.375234 | controller |     - '    logger.go:42: 09:34:30 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.375240 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.375249 | controller |       found'
2026-03-14 09:44:27.375255 | controller |     - '    logger.go:42: 09:34:31 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.375261 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.375266 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.375272 | controller |     - '        '
2026-03-14 09:44:27.375278 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.375283 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.375312 | controller |     - '        '
2026-03-14 09:44:27.375319 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.375325 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.375332 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.375339 | controller |     - '          exit 1'
2026-03-14 09:44:27.375344 | controller |     - '        fi'
2026-03-14 09:44:27.375351 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.375361 | controller |     - '        '
2026-03-14 09:44:27.375366 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.375372 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.375378 | controller |       | base64 -d)'
2026-03-14 09:44:27.375384 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.375391 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.375397 | controller |     - '          exit 1'
2026-03-14 09:44:27.375403 | controller |     - '        fi'
2026-03-14 09:44:27.375409 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.375415 | controller |     - '        '
2026-03-14 09:44:27.375421 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.375430 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.375437 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.375443 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.375449 | controller |       --timeout=300s'
2026-03-14 09:44:27.375455 | controller |     - '        '
2026-03-14 09:44:27.375460 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.375466 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.375472 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.375477 | controller |     - '        '
2026-03-14 09:44:27.375483 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.375492 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.375498 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.375504 | controller |     - '        '
2026-03-14 09:44:27.375510 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.375516 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.375522 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.375529 | controller |       available"'
2026-03-14 09:44:27.375545 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.375553 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.375559 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.375581 | controller |     - '          exit 1'
2026-03-14 09:44:27.375588 | controller |     - '        fi'
2026-03-14 09:44:27.375594 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.375600 | controller |     - '        '
2026-03-14 09:44:27.375606 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.375612 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.375618 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.375624 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.375630 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.375636 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.375642 | controller |       -- \'
2026-03-14 09:44:27.375648 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.375654 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.375660 | controller |     - '    logger.go:42: 09:34:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.375666 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.375673 | controller |     - '    logger.go:42: 09:34:31 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.375679 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.375685 | controller |       found'
2026-03-14 09:44:27.375692 | controller |     - '    logger.go:42: 09:34:32 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.375698 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.375704 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.375710 | controller |     - '        '
2026-03-14 09:44:27.375717 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.375724 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.375730 | controller |     - '        '
2026-03-14 09:44:27.375737 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.375744 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.375750 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.375756 | controller |     - '          exit 1'
2026-03-14 09:44:27.375763 | controller |     - '        fi'
2026-03-14 09:44:27.375769 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.375776 | controller |     - '        '
2026-03-14 09:44:27.375782 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.375788 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.375794 | controller |       | base64 -d)'
2026-03-14 09:44:27.375801 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.375807 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.375814 | controller |     - '          exit 1'
2026-03-14 09:44:27.375820 | controller |     - '        fi'
2026-03-14 09:44:27.375826 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.375832 | controller |     - '        '
2026-03-14 09:44:27.375839 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.375845 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.375851 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.375857 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.375878 | controller |       --timeout=300s'
2026-03-14 09:44:27.375890 | controller |     - '        '
2026-03-14 09:44:27.375896 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.375902 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.375907 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.375913 | controller |     - '        '
2026-03-14 09:44:27.375919 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.375925 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.375930 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.375936 | controller |     - '        '
2026-03-14 09:44:27.375942 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.375947 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.375953 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.375959 | controller |       available"'
2026-03-14 09:44:27.375965 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.375981 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.375988 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.375995 | controller |     - '          exit 1'
2026-03-14 09:44:27.376002 | controller |     - '        fi'
2026-03-14 09:44:27.376008 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.376014 | controller |     - '        '
2026-03-14 09:44:27.376020 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.376026 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.376032 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.376038 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.376044 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.376050 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.376056 | controller |       -- \'
2026-03-14 09:44:27.376061 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.376067 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.376073 | controller |     - '    logger.go:42: 09:34:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.376079 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.376085 | controller |     - '    logger.go:42: 09:34:33 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.376090 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.376096 | controller |       found'
2026-03-14 09:44:27.376102 | controller |     - '    logger.go:42: 09:34:34 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.376108 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.376114 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.376120 | controller |     - '        '
2026-03-14 09:44:27.376126 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.376131 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.376150 | controller |     - '        '
2026-03-14 09:44:27.376157 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.376163 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.376169 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.376174 | controller |     - '          exit 1'
2026-03-14 09:44:27.376180 | controller |     - '        fi'
2026-03-14 09:44:27.376187 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.376192 | controller |     - '        '
2026-03-14 09:44:27.376198 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.376204 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.376209 | controller |       | base64 -d)'
2026-03-14 09:44:27.376215 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.376221 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.376227 | controller |     - '          exit 1'
2026-03-14 09:44:27.376233 | controller |     - '        fi'
2026-03-14 09:44:27.376239 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.376246 | controller |     - '        '
2026-03-14 09:44:27.376252 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.376258 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.376268 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.376275 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.376280 | controller |       --timeout=300s'
2026-03-14 09:44:27.376304 | controller |     - '        '
2026-03-14 09:44:27.376316 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.376323 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.376330 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.376336 | controller |     - '        '
2026-03-14 09:44:27.376343 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.376349 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.376356 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.376363 | controller |     - '        '
2026-03-14 09:44:27.376370 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.376376 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.376382 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.376388 | controller |       available"'
2026-03-14 09:44:27.376394 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.376412 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.376417 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.376422 | controller |     - '          exit 1'
2026-03-14 09:44:27.376426 | controller |     - '        fi'
2026-03-14 09:44:27.376430 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.376435 | controller |     - '        '
2026-03-14 09:44:27.376439 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.376443 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.376447 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.376462 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.376466 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.376470 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.376475 | controller |       -- \'
2026-03-14 09:44:27.376479 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.376483 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.376487 | controller |     - '    logger.go:42: 09:34:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.376491 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.376496 | controller |     - '    logger.go:42: 09:34:34 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.376500 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.376504 | controller |       found'
2026-03-14 09:44:27.376508 | controller |     - '    logger.go:42: 09:34:35 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.376512 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.376516 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.376521 | controller |     - '        '
2026-03-14 09:44:27.376525 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.376529 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.376533 | controller |     - '        '
2026-03-14 09:44:27.376537 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.376541 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.376546 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.376550 | controller |     - '          exit 1'
2026-03-14 09:44:27.376554 | controller |     - '        fi'
2026-03-14 09:44:27.376558 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.376563 | controller |     - '        '
2026-03-14 09:44:27.376567 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.376571 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.376575 | controller |       | base64 -d)'
2026-03-14 09:44:27.376579 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.376584 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.376588 | controller |     - '          exit 1'
2026-03-14 09:44:27.376592 | controller |     - '        fi'
2026-03-14 09:44:27.376596 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.376600 | controller |     - '        '
2026-03-14 09:44:27.376604 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.376609 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.376613 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.376619 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.376624 | controller |       --timeout=300s'
2026-03-14 09:44:27.376630 | controller |     - '        '
2026-03-14 09:44:27.376647 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.376653 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.376658 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.376664 | controller |     - '        '
2026-03-14 09:44:27.376672 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.376678 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.376684 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.376689 | controller |     - '        '
2026-03-14 09:44:27.376695 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.376700 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.376706 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.376712 | controller |       available"'
2026-03-14 09:44:27.376718 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.376722 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.376734 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.376738 | controller |     - '          exit 1'
2026-03-14 09:44:27.376742 | controller |     - '        fi'
2026-03-14 09:44:27.376747 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.376751 | controller |     - '        '
2026-03-14 09:44:27.376755 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.376759 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.376763 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.376767 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.376772 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.376776 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.376783 | controller |       -- \'
2026-03-14 09:44:27.376787 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.376791 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.376795 | controller |     - '    logger.go:42: 09:34:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.376800 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.376804 | controller |     - '    logger.go:42: 09:34:35 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.376808 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.376812 | controller |       found'
2026-03-14 09:44:27.376817 | controller |     - '    logger.go:42: 09:34:36 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.376822 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.376828 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.376833 | controller |     - '        '
2026-03-14 09:44:27.376839 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.376844 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.376850 | controller |     - '        '
2026-03-14 09:44:27.376855 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.376861 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.376866 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.376870 | controller |     - '          exit 1'
2026-03-14 09:44:27.376875 | controller |     - '        fi'
2026-03-14 09:44:27.376879 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.376883 | controller |     - '        '
2026-03-14 09:44:27.376890 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.376894 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.376898 | controller |       | base64 -d)'
2026-03-14 09:44:27.376903 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.376907 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.376911 | controller |     - '          exit 1'
2026-03-14 09:44:27.376915 | controller |     - '        fi'
2026-03-14 09:44:27.376919 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.376924 | controller |     - '        '
2026-03-14 09:44:27.376928 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.376932 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.376936 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.376943 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.376947 | controller |       --timeout=300s'
2026-03-14 09:44:27.376951 | controller |     - '        '
2026-03-14 09:44:27.376955 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.376959 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.376964 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.376968 | controller |     - '        '
2026-03-14 09:44:27.376972 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.376976 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.376981 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.376985 | controller |     - '        '
2026-03-14 09:44:27.376989 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.376993 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.376997 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.377002 | controller |       available"'
2026-03-14 09:44:27.377006 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.377010 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.377014 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.377025 | controller |     - '          exit 1'
2026-03-14 09:44:27.377029 | controller |     - '        fi'
2026-03-14 09:44:27.377034 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.377038 | controller |     - '        '
2026-03-14 09:44:27.377042 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.377046 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.377051 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.377055 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.377059 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.377063 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.377067 | controller |       -- \'
2026-03-14 09:44:27.377072 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.377078 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.377084 | controller |     - '    logger.go:42: 09:34:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.377088 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.377092 | controller |     - '    logger.go:42: 09:34:36 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.377097 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.377101 | controller |       found'
2026-03-14 09:44:27.377105 | controller |     - '    logger.go:42: 09:34:37 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.377109 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.377113 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.377118 | controller |     - '        '
2026-03-14 09:44:27.377122 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.377126 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.377130 | controller |     - '        '
2026-03-14 09:44:27.377134 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.377139 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.377143 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.377147 | controller |     - '          exit 1'
2026-03-14 09:44:27.377151 | controller |     - '        fi'
2026-03-14 09:44:27.377156 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.377160 | controller |     - '        '
2026-03-14 09:44:27.377164 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.377168 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.377172 | controller |       | base64 -d)'
2026-03-14 09:44:27.377177 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.377181 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.377185 | controller |     - '          exit 1'
2026-03-14 09:44:27.377189 | controller |     - '        fi'
2026-03-14 09:44:27.377193 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.377198 | controller |     - '        '
2026-03-14 09:44:27.377202 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.377206 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.377210 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.377214 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.377219 | controller |       --timeout=300s'
2026-03-14 09:44:27.377223 | controller |     - '        '
2026-03-14 09:44:27.377227 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.377231 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.377235 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.377240 | controller |     - '        '
2026-03-14 09:44:27.377244 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.377248 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.377252 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.377258 | controller |     - '        '
2026-03-14 09:44:27.377263 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.377267 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.377274 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.377279 | controller |       available"'
2026-03-14 09:44:27.377283 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.377302 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.377310 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.377315 | controller |     - '          exit 1'
2026-03-14 09:44:27.377320 | controller |     - '        fi'
2026-03-14 09:44:27.377330 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.377335 | controller |     - '        '
2026-03-14 09:44:27.377339 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.377343 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.377348 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.377352 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.377356 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.377361 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.377365 | controller |       -- \'
2026-03-14 09:44:27.377369 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.377373 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.377378 | controller |     - '    logger.go:42: 09:34:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.377382 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.377386 | controller |     - '    logger.go:42: 09:34:37 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.377390 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.377395 | controller |       found'
2026-03-14 09:44:27.377399 | controller |     - '    logger.go:42: 09:34:38 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.377403 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.377407 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.377412 | controller |     - '        '
2026-03-14 09:44:27.377416 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.377420 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.377424 | controller |     - '        '
2026-03-14 09:44:27.377429 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.377433 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.377437 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.377442 | controller |     - '          exit 1'
2026-03-14 09:44:27.377446 | controller |     - '        fi'
2026-03-14 09:44:27.377450 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.377455 | controller |     - '        '
2026-03-14 09:44:27.377459 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.377463 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.377467 | controller |       | base64 -d)'
2026-03-14 09:44:27.377472 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.377476 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.377480 | controller |     - '          exit 1'
2026-03-14 09:44:27.377484 | controller |     - '        fi'
2026-03-14 09:44:27.377489 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.377495 | controller |     - '        '
2026-03-14 09:44:27.377500 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.377504 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.377508 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.377512 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.377517 | controller |       --timeout=300s'
2026-03-14 09:44:27.377521 | controller |     - '        '
2026-03-14 09:44:27.377525 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.377530 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.377534 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.377538 | controller |     - '        '
2026-03-14 09:44:27.377542 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.377547 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.377551 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.377555 | controller |     - '        '
2026-03-14 09:44:27.377569 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.377575 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.377579 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.377583 | controller |       available"'
2026-03-14 09:44:27.377588 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.377592 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.377596 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.377600 | controller |     - '          exit 1'
2026-03-14 09:44:27.377605 | controller |     - '        fi'
2026-03-14 09:44:27.377614 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.377620 | controller |     - '        '
2026-03-14 09:44:27.377626 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.377632 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.377638 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.377644 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.377650 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.377656 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.377662 | controller |       -- \'
2026-03-14 09:44:27.377667 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.377673 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.377679 | controller |     - '    logger.go:42: 09:34:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.377684 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.377690 | controller |     - '    logger.go:42: 09:34:38 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.377695 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.377711 | controller |       found'
2026-03-14 09:44:27.377718 | controller |     - '    logger.go:42: 09:34:39 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.377722 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.377729 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.377733 | controller |     - '        '
2026-03-14 09:44:27.377737 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.377742 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.377746 | controller |     - '        '
2026-03-14 09:44:27.377751 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.377757 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.377763 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.377769 | controller |     - '          exit 1'
2026-03-14 09:44:27.377774 | controller |     - '        fi'
2026-03-14 09:44:27.377780 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.377786 | controller |     - '        '
2026-03-14 09:44:27.377795 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.377801 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.377805 | controller |       | base64 -d)'
2026-03-14 09:44:27.377811 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.377815 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.377821 | controller |     - '          exit 1'
2026-03-14 09:44:27.377827 | controller |     - '        fi'
2026-03-14 09:44:27.377833 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.377838 | controller |     - '        '
2026-03-14 09:44:27.377844 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.377850 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.377856 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.377862 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.377867 | controller |       --timeout=300s'
2026-03-14 09:44:27.377873 | controller |     - '        '
2026-03-14 09:44:27.377878 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.377883 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.377887 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.377892 | controller |     - '        '
2026-03-14 09:44:27.377898 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.377904 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.377913 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.377919 | controller |     - '        '
2026-03-14 09:44:27.377925 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.377932 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.377938 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.377945 | controller |       available"'
2026-03-14 09:44:27.377952 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.377958 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.377964 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.377969 | controller |     - '          exit 1'
2026-03-14 09:44:27.377978 | controller |     - '        fi'
2026-03-14 09:44:27.377984 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.377990 | controller |     - '        '
2026-03-14 09:44:27.378006 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.378013 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.378019 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.378025 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.378031 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.378037 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.378043 | controller |       -- \'
2026-03-14 09:44:27.378049 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.378054 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.378061 | controller |     - '    logger.go:42: 09:34:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.378067 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.378095 | controller |     - '    logger.go:42: 09:34:40 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.378102 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.378108 | controller |       found'
2026-03-14 09:44:27.378114 | controller |     - '    logger.go:42: 09:34:41 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.378120 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.378126 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.378132 | controller |     - '        '
2026-03-14 09:44:27.378138 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.378143 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.378149 | controller |     - '        '
2026-03-14 09:44:27.378155 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.378160 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.378166 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.378172 | controller |     - '          exit 1'
2026-03-14 09:44:27.378177 | controller |     - '        fi'
2026-03-14 09:44:27.378183 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.378188 | controller |     - '        '
2026-03-14 09:44:27.378194 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.378200 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.378206 | controller |       | base64 -d)'
2026-03-14 09:44:27.378211 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.378217 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.378223 | controller |     - '          exit 1'
2026-03-14 09:44:27.378229 | controller |     - '        fi'
2026-03-14 09:44:27.378234 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.378240 | controller |     - '        '
2026-03-14 09:44:27.378246 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.378251 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.378257 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.378263 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.378269 | controller |       --timeout=300s'
2026-03-14 09:44:27.378278 | controller |     - '        '
2026-03-14 09:44:27.378285 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.378307 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.378316 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.378322 | controller |     - '        '
2026-03-14 09:44:27.378328 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.378334 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.378339 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.378345 | controller |     - '        '
2026-03-14 09:44:27.378351 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.378357 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.378375 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.378381 | controller |       available"'
2026-03-14 09:44:27.378385 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.378390 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.378394 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.378398 | controller |     - '          exit 1'
2026-03-14 09:44:27.378402 | controller |     - '        fi'
2026-03-14 09:44:27.378407 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.378411 | controller |     - '        '
2026-03-14 09:44:27.378423 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.378427 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.378432 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.378436 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.378440 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.378444 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.378448 | controller |       -- \'
2026-03-14 09:44:27.378453 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.378457 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.378461 | controller |     - '    logger.go:42: 09:34:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.378465 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.378470 | controller |     - '    logger.go:42: 09:34:41 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.378474 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.378478 | controller |       found'
2026-03-14 09:44:27.378482 | controller |     - '    logger.go:42: 09:34:42 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.378486 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.378491 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.378495 | controller |     - '        '
2026-03-14 09:44:27.378499 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.378503 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.378507 | controller |     - '        '
2026-03-14 09:44:27.378520 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.378525 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.378529 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.378534 | controller |     - '          exit 1'
2026-03-14 09:44:27.378538 | controller |     - '        fi'
2026-03-14 09:44:27.378542 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.378546 | controller |     - '        '
2026-03-14 09:44:27.378550 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.378555 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.378559 | controller |       | base64 -d)'
2026-03-14 09:44:27.378563 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.378567 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.378571 | controller |     - '          exit 1'
2026-03-14 09:44:27.378575 | controller |     - '        fi'
2026-03-14 09:44:27.378580 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.378584 | controller |     - '        '
2026-03-14 09:44:27.378588 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.378592 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.378597 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.378601 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.378605 | controller |       --timeout=300s'
2026-03-14 09:44:27.378609 | controller |     - '        '
2026-03-14 09:44:27.378614 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.378620 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.378626 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.378632 | controller |     - '        '
2026-03-14 09:44:27.378641 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.378647 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.378651 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.378655 | controller |     - '        '
2026-03-14 09:44:27.378659 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.378664 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.378668 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.378672 | controller |       available"'
2026-03-14 09:44:27.378676 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.378680 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.378685 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.378689 | controller |     - '          exit 1'
2026-03-14 09:44:27.378693 | controller |     - '        fi'
2026-03-14 09:44:27.378697 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.378701 | controller |     - '        '
2026-03-14 09:44:27.378706 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.378717 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.378721 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.378728 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.378732 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.378736 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.378741 | controller |       -- \'
2026-03-14 09:44:27.378745 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.378749 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.378753 | controller |     - '    logger.go:42: 09:34:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.378757 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.378762 | controller |     - '    logger.go:42: 09:34:42 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.378766 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.378770 | controller |       found'
2026-03-14 09:44:27.378774 | controller |     - '    logger.go:42: 09:34:43 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.378779 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.378783 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.378787 | controller |     - '        '
2026-03-14 09:44:27.378791 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.378795 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.378799 | controller |     - '        '
2026-03-14 09:44:27.378804 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.378808 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.378812 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.378816 | controller |     - '          exit 1'
2026-03-14 09:44:27.378820 | controller |     - '        fi'
2026-03-14 09:44:27.378825 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.378829 | controller |     - '        '
2026-03-14 09:44:27.378833 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.378837 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.378842 | controller |       | base64 -d)'
2026-03-14 09:44:27.378846 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.378850 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.378854 | controller |     - '          exit 1'
2026-03-14 09:44:27.378858 | controller |     - '        fi'
2026-03-14 09:44:27.378863 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.378867 | controller |     - '        '
2026-03-14 09:44:27.378871 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.378875 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.378880 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.378884 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.378888 | controller |       --timeout=300s'
2026-03-14 09:44:27.378892 | controller |     - '        '
2026-03-14 09:44:27.378897 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.378901 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.378905 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.378910 | controller |     - '        '
2026-03-14 09:44:27.378914 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.378920 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.378924 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.378928 | controller |     - '        '
2026-03-14 09:44:27.378933 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.378937 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.378941 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.378945 | controller |       available"'
2026-03-14 09:44:27.378950 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.378957 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.378962 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.378966 | controller |     - '          exit 1'
2026-03-14 09:44:27.378970 | controller |     - '        fi'
2026-03-14 09:44:27.378974 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.378979 | controller |     - '        '
2026-03-14 09:44:27.378983 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.378987 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.378998 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.379002 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.379007 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.379011 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.379015 | controller |       -- \'
2026-03-14 09:44:27.379020 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.379024 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.379038 | controller |     - '    logger.go:42: 09:34:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.379043 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.379048 | controller |     - '    logger.go:42: 09:34:43 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.379052 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.379056 | controller |       found'
2026-03-14 09:44:27.379061 | controller |     - '    logger.go:42: 09:34:44 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.379065 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.379069 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.379074 | controller |     - '        '
2026-03-14 09:44:27.379078 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.379084 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.379088 | controller |     - '        '
2026-03-14 09:44:27.379093 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.379097 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.379101 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.379106 | controller |     - '          exit 1'
2026-03-14 09:44:27.379110 | controller |     - '        fi'
2026-03-14 09:44:27.379114 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.379119 | controller |     - '        '
2026-03-14 09:44:27.379123 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.379129 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.379133 | controller |       | base64 -d)'
2026-03-14 09:44:27.379138 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.379142 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.379146 | controller |     - '          exit 1'
2026-03-14 09:44:27.379151 | controller |     - '        fi'
2026-03-14 09:44:27.379155 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.379159 | controller |     - '        '
2026-03-14 09:44:27.379163 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.379168 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.379172 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.379176 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.379181 | controller |       --timeout=300s'
2026-03-14 09:44:27.379185 | controller |     - '        '
2026-03-14 09:44:27.379189 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.379193 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.379198 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.379202 | controller |     - '        '
2026-03-14 09:44:27.379206 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.379211 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.379215 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.379219 | controller |     - '        '
2026-03-14 09:44:27.379223 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.379228 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.379232 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.379236 | controller |       available"'
2026-03-14 09:44:27.379241 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.379247 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.379252 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.379256 | controller |     - '          exit 1'
2026-03-14 09:44:27.379260 | controller |     - '        fi'
2026-03-14 09:44:27.379265 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.379269 | controller |     - '        '
2026-03-14 09:44:27.379273 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.379278 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.379290 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.379309 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.379313 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.379318 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.379322 | controller |       -- \'
2026-03-14 09:44:27.379326 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.379331 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.379337 | controller |     - '    logger.go:42: 09:34:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.379342 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.379346 | controller |     - '    logger.go:42: 09:34:44 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.379350 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.379354 | controller |       found'
2026-03-14 09:44:27.379359 | controller |     - '    logger.go:42: 09:34:45 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.379363 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.379367 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.379371 | controller |     - '        '
2026-03-14 09:44:27.379376 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.379381 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.379385 | controller |     - '        '
2026-03-14 09:44:27.379390 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.379394 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.379398 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.379402 | controller |     - '          exit 1'
2026-03-14 09:44:27.379407 | controller |     - '        fi'
2026-03-14 09:44:27.379411 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.379415 | controller |     - '        '
2026-03-14 09:44:27.379420 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.379424 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.379428 | controller |       | base64 -d)'
2026-03-14 09:44:27.379432 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.379437 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.379441 | controller |     - '          exit 1'
2026-03-14 09:44:27.379445 | controller |     - '        fi'
2026-03-14 09:44:27.379450 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.379454 | controller |     - '        '
2026-03-14 09:44:27.379458 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.379462 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.379467 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.379471 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.379475 | controller |       --timeout=300s'
2026-03-14 09:44:27.379480 | controller |     - '        '
2026-03-14 09:44:27.379484 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.379488 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.379492 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.379497 | controller |     - '        '
2026-03-14 09:44:27.379501 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.379505 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.379509 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.379514 | controller |     - '        '
2026-03-14 09:44:27.379518 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.379522 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.379528 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.379532 | controller |       available"'
2026-03-14 09:44:27.379537 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.379541 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.379545 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.379550 | controller |     - '          exit 1'
2026-03-14 09:44:27.379554 | controller |     - '        fi'
2026-03-14 09:44:27.379558 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.379562 | controller |     - '        '
2026-03-14 09:44:27.379569 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.379574 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.379578 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.379588 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.379593 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.379597 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.379602 | controller |       -- \'
2026-03-14 09:44:27.379606 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.379610 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.379615 | controller |     - '    logger.go:42: 09:34:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.379621 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.379627 | controller |     - '    logger.go:42: 09:34:45 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.379633 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.379639 | controller |       found'
2026-03-14 09:44:27.379645 | controller |     - '    logger.go:42: 09:34:46 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.379652 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.379658 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.379663 | controller |     - '        '
2026-03-14 09:44:27.379669 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.379675 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.379680 | controller |     - '        '
2026-03-14 09:44:27.379686 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.379691 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.379697 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.379701 | controller |     - '          exit 1'
2026-03-14 09:44:27.379706 | controller |     - '        fi'
2026-03-14 09:44:27.379710 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.379714 | controller |     - '        '
2026-03-14 09:44:27.379718 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.379723 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.379727 | controller |       | base64 -d)'
2026-03-14 09:44:27.379731 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.379736 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.379740 | controller |     - '          exit 1'
2026-03-14 09:44:27.379744 | controller |     - '        fi'
2026-03-14 09:44:27.379748 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.379756 | controller |     - '        '
2026-03-14 09:44:27.379760 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.379764 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.379768 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.379773 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.379777 | controller |       --timeout=300s'
2026-03-14 09:44:27.379781 | controller |     - '        '
2026-03-14 09:44:27.379785 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.379790 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.379794 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.379798 | controller |     - '        '
2026-03-14 09:44:27.379803 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.379807 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.379811 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.379816 | controller |     - '        '
2026-03-14 09:44:27.379820 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.379824 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.379828 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.379833 | controller |       available"'
2026-03-14 09:44:27.379837 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.379841 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.379845 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.379850 | controller |     - '          exit 1'
2026-03-14 09:44:27.379854 | controller |     - '        fi'
2026-03-14 09:44:27.379858 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.379862 | controller |     - '        '
2026-03-14 09:44:27.379867 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.379871 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.379875 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.379886 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.379894 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.379898 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.379903 | controller |       -- \'
2026-03-14 09:44:27.379907 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.379911 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.379916 | controller |     - '    logger.go:42: 09:34:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.379920 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.379925 | controller |     - '    logger.go:42: 09:34:46 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.379931 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.379935 | controller |       found'
2026-03-14 09:44:27.379940 | controller |     - '    logger.go:42: 09:34:47 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.379947 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.379952 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.379957 | controller |     - '        '
2026-03-14 09:44:27.379961 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.379965 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.379969 | controller |     - '        '
2026-03-14 09:44:27.379974 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.379978 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.379982 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.379987 | controller |     - '          exit 1'
2026-03-14 09:44:27.379991 | controller |     - '        fi'
2026-03-14 09:44:27.379995 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.379999 | controller |     - '        '
2026-03-14 09:44:27.380004 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.380008 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.380012 | controller |       | base64 -d)'
2026-03-14 09:44:27.380016 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.380023 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.380027 | controller |     - '          exit 1'
2026-03-14 09:44:27.380033 | controller |     - '        fi'
2026-03-14 09:44:27.380039 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.380045 | controller |     - '        '
2026-03-14 09:44:27.380051 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.380057 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.380062 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.380068 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.380073 | controller |       --timeout=300s'
2026-03-14 09:44:27.380077 | controller |     - '        '
2026-03-14 09:44:27.380082 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.380086 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.380090 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.380094 | controller |     - '        '
2026-03-14 09:44:27.380099 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.380103 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.380108 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.380113 | controller |     - '        '
2026-03-14 09:44:27.380122 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.380130 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.380136 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.380142 | controller |       available"'
2026-03-14 09:44:27.380148 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.380154 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.380160 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.380166 | controller |     - '          exit 1'
2026-03-14 09:44:27.380172 | controller |     - '        fi'
2026-03-14 09:44:27.380178 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.380187 | controller |     - '        '
2026-03-14 09:44:27.380194 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.380200 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.380206 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.380216 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.380229 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.380233 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.380238 | controller |       -- \'
2026-03-14 09:44:27.380242 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.380247 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.380251 | controller |     - '    logger.go:42: 09:34:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.380255 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.380260 | controller |     - '    logger.go:42: 09:34:48 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.380264 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.380268 | controller |       found'
2026-03-14 09:44:27.380274 | controller |     - '    logger.go:42: 09:34:49 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.380280 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.380286 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.380312 | controller |     - '        '
2026-03-14 09:44:27.380318 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.380324 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.380330 | controller |     - '        '
2026-03-14 09:44:27.380335 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.380341 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.380347 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.380353 | controller |     - '          exit 1'
2026-03-14 09:44:27.380358 | controller |     - '        fi'
2026-03-14 09:44:27.380364 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.380370 | controller |     - '        '
2026-03-14 09:44:27.380376 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.380381 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.380387 | controller |       | base64 -d)'
2026-03-14 09:44:27.380393 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.380398 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.380404 | controller |     - '          exit 1'
2026-03-14 09:44:27.380410 | controller |     - '        fi'
2026-03-14 09:44:27.380416 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.380422 | controller |     - '        '
2026-03-14 09:44:27.380428 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.380433 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.380439 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.380445 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.380451 | controller |       --timeout=300s'
2026-03-14 09:44:27.380460 | controller |     - '        '
2026-03-14 09:44:27.380466 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.380475 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.380481 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.380487 | controller |     - '        '
2026-03-14 09:44:27.380493 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.380498 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.380504 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.380510 | controller |     - '        '
2026-03-14 09:44:27.380516 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.380522 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.380527 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.380533 | controller |       available"'
2026-03-14 09:44:27.380539 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.380545 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.380550 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.380556 | controller |     - '          exit 1'
2026-03-14 09:44:27.380562 | controller |     - '        fi'
2026-03-14 09:44:27.380567 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.380571 | controller |     - '        '
2026-03-14 09:44:27.380575 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.380579 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.380584 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.380588 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.380601 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.380607 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.380613 | controller |       -- \'
2026-03-14 09:44:27.380619 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.380625 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.380631 | controller |     - '    logger.go:42: 09:34:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.380637 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.380642 | controller |     - '    logger.go:42: 09:34:49 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.380648 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.380654 | controller |       found'
2026-03-14 09:44:27.380660 | controller |     - '    logger.go:42: 09:34:50 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.380665 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.380671 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.380677 | controller |     - '        '
2026-03-14 09:44:27.380683 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.380688 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.380694 | controller |     - '        '
2026-03-14 09:44:27.380700 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.380709 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.380715 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.380721 | controller |     - '          exit 1'
2026-03-14 09:44:27.380727 | controller |     - '        fi'
2026-03-14 09:44:27.380732 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.380738 | controller |     - '        '
2026-03-14 09:44:27.380747 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.380753 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.380759 | controller |       | base64 -d)'
2026-03-14 09:44:27.380764 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.380770 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.380776 | controller |     - '          exit 1'
2026-03-14 09:44:27.380782 | controller |     - '        fi'
2026-03-14 09:44:27.380788 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.380794 | controller |     - '        '
2026-03-14 09:44:27.380799 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.380805 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.380811 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.380817 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.380822 | controller |       --timeout=300s'
2026-03-14 09:44:27.380828 | controller |     - '        '
2026-03-14 09:44:27.380834 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.380840 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.380846 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.380851 | controller |     - '        '
2026-03-14 09:44:27.380857 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.380863 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.380869 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.380874 | controller |     - '        '
2026-03-14 09:44:27.380880 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.380886 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.380892 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.380896 | controller |       available"'
2026-03-14 09:44:27.380900 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.380905 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.380909 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.380913 | controller |     - '          exit 1'
2026-03-14 09:44:27.380917 | controller |     - '        fi'
2026-03-14 09:44:27.380923 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.380928 | controller |     - '        '
2026-03-14 09:44:27.380933 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.380937 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.380942 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.380947 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.380955 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.380969 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.380974 | controller |       -- \'
2026-03-14 09:44:27.380978 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.380983 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.380987 | controller |     - '    logger.go:42: 09:34:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.380991 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.380995 | controller |     - '    logger.go:42: 09:34:50 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.381000 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.381004 | controller |       found'
2026-03-14 09:44:27.381008 | controller |     - '    logger.go:42: 09:34:51 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.381012 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.381016 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.381021 | controller |     - '        '
2026-03-14 09:44:27.381025 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.381029 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.381033 | controller |     - '        '
2026-03-14 09:44:27.381037 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.381042 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.381046 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.381050 | controller |     - '          exit 1'
2026-03-14 09:44:27.381054 | controller |     - '        fi'
2026-03-14 09:44:27.381059 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.381063 | controller |     - '        '
2026-03-14 09:44:27.381067 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.381071 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.381075 | controller |       | base64 -d)'
2026-03-14 09:44:27.381080 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.381084 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.381088 | controller |     - '          exit 1'
2026-03-14 09:44:27.381092 | controller |     - '        fi'
2026-03-14 09:44:27.381096 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.381101 | controller |     - '        '
2026-03-14 09:44:27.381105 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.381109 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.381113 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.381117 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.381122 | controller |       --timeout=300s'
2026-03-14 09:44:27.381126 | controller |     - '        '
2026-03-14 09:44:27.381130 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.381134 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.381138 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.381143 | controller |     - '        '
2026-03-14 09:44:27.381147 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.381153 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.381157 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.381161 | controller |     - '        '
2026-03-14 09:44:27.381166 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.381170 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.381174 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.381178 | controller |       available"'
2026-03-14 09:44:27.381182 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.381187 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.381191 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.381195 | controller |     - '          exit 1'
2026-03-14 09:44:27.381199 | controller |     - '        fi'
2026-03-14 09:44:27.381203 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.381208 | controller |     - '        '
2026-03-14 09:44:27.381212 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.381216 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.381220 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.381224 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.381229 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.381238 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.381242 | controller |       -- \'
2026-03-14 09:44:27.381246 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.381251 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.381255 | controller |     - '    logger.go:42: 09:34:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.381259 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.381263 | controller |     - '    logger.go:42: 09:34:51 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.381267 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.381272 | controller |       found'
2026-03-14 09:44:27.381276 | controller |     - '    logger.go:42: 09:34:52 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.381280 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.381284 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.381303 | controller |     - '        '
2026-03-14 09:44:27.381311 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.381315 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.381320 | controller |     - '        '
2026-03-14 09:44:27.381324 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.381328 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.381333 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.381337 | controller |     - '          exit 1'
2026-03-14 09:44:27.381341 | controller |     - '        fi'
2026-03-14 09:44:27.381345 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.381350 | controller |     - '        '
2026-03-14 09:44:27.381354 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.381358 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.381365 | controller |       | base64 -d)'
2026-03-14 09:44:27.381369 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.381373 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.381377 | controller |     - '          exit 1'
2026-03-14 09:44:27.381382 | controller |     - '        fi'
2026-03-14 09:44:27.381388 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.381392 | controller |     - '        '
2026-03-14 09:44:27.381396 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.381401 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.381405 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.381409 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.381413 | controller |       --timeout=300s'
2026-03-14 09:44:27.381418 | controller |     - '        '
2026-03-14 09:44:27.381422 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.381426 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.381440 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.381444 | controller |     - '        '
2026-03-14 09:44:27.381448 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.381453 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.381457 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.381461 | controller |     - '        '
2026-03-14 09:44:27.381465 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.381469 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.381474 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.381478 | controller |       available"'
2026-03-14 09:44:27.381482 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.381486 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.381491 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.381495 | controller |     - '          exit 1'
2026-03-14 09:44:27.381499 | controller |     - '        fi'
2026-03-14 09:44:27.381503 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.381620 | controller |     - '        '
2026-03-14 09:44:27.381630 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.381636 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.381642 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.381648 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.381654 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.381659 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.381663 | controller |       -- \'
2026-03-14 09:44:27.381675 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.381680 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.381689 | controller |     - '    logger.go:42: 09:34:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.381694 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.381699 | controller |     - '    logger.go:42: 09:34:52 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.381704 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.381708 | controller |       found'
2026-03-14 09:44:27.381712 | controller |     - '    logger.go:42: 09:34:53 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.381716 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.381721 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.381725 | controller |     - '        '
2026-03-14 09:44:27.381729 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.381733 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.381738 | controller |     - '        '
2026-03-14 09:44:27.381742 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.381746 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.381752 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.381756 | controller |     - '          exit 1'
2026-03-14 09:44:27.381761 | controller |     - '        fi'
2026-03-14 09:44:27.381765 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.381769 | controller |     - '        '
2026-03-14 09:44:27.381773 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.381778 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.381782 | controller |       | base64 -d)'
2026-03-14 09:44:27.381786 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.381790 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.381794 | controller |     - '          exit 1'
2026-03-14 09:44:27.381799 | controller |     - '        fi'
2026-03-14 09:44:27.381803 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.381807 | controller |     - '        '
2026-03-14 09:44:27.381811 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.381816 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.381820 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.381824 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.381828 | controller |       --timeout=300s'
2026-03-14 09:44:27.381832 | controller |     - '        '
2026-03-14 09:44:27.381837 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.381841 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.381845 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.381849 | controller |     - '        '
2026-03-14 09:44:27.381854 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.381858 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.381862 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.381866 | controller |     - '        '
2026-03-14 09:44:27.381870 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.381875 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.381879 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.381886 | controller |       available"'
2026-03-14 09:44:27.381890 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.381895 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.381899 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.381903 | controller |     - '          exit 1'
2026-03-14 09:44:27.381907 | controller |     - '        fi'
2026-03-14 09:44:27.381912 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.381916 | controller |     - '        '
2026-03-14 09:44:27.381920 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.381924 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.381929 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.381933 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.381937 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.381941 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.381946 | controller |       -- \'
2026-03-14 09:44:27.381956 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.381960 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.381965 | controller |     - '    logger.go:42: 09:34:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.381969 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.381973 | controller |     - '    logger.go:42: 09:34:53 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.381978 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.381982 | controller |       found'
2026-03-14 09:44:27.381986 | controller |     - '    logger.go:42: 09:34:54 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.381990 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.381996 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.382002 | controller |     - '        '
2026-03-14 09:44:27.382008 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.382014 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.382020 | controller |     - '        '
2026-03-14 09:44:27.382025 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.382031 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.382037 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.382043 | controller |     - '          exit 1'
2026-03-14 09:44:27.382048 | controller |     - '        fi'
2026-03-14 09:44:27.382054 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.382059 | controller |     - '        '
2026-03-14 09:44:27.382065 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.382071 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.382076 | controller |       | base64 -d)'
2026-03-14 09:44:27.382106 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.382114 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.382120 | controller |     - '          exit 1'
2026-03-14 09:44:27.382126 | controller |     - '        fi'
2026-03-14 09:44:27.382132 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.382138 | controller |     - '        '
2026-03-14 09:44:27.382143 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.382150 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.382155 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.382159 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.382163 | controller |       --timeout=300s'
2026-03-14 09:44:27.382167 | controller |     - '        '
2026-03-14 09:44:27.382171 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.382176 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.382180 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.382184 | controller |     - '        '
2026-03-14 09:44:27.382188 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.382192 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.382197 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.382201 | controller |     - '        '
2026-03-14 09:44:27.382205 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.382209 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.382213 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.382218 | controller |       available"'
2026-03-14 09:44:27.382222 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.382229 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.382233 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.382237 | controller |     - '          exit 1'
2026-03-14 09:44:27.382241 | controller |     - '        fi'
2026-03-14 09:44:27.382245 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.382250 | controller |     - '        '
2026-03-14 09:44:27.382254 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.382258 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.382262 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.382266 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.382271 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.382275 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.382279 | controller |       -- \'
2026-03-14 09:44:27.382303 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.382311 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.382315 | controller |     - '    logger.go:42: 09:34:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.382320 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.382324 | controller |     - '    logger.go:42: 09:34:54 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.382330 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.382334 | controller |       found'
2026-03-14 09:44:27.382338 | controller |     - '    logger.go:42: 09:34:56 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.382344 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.382348 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.382353 | controller |     - '        '
2026-03-14 09:44:27.382357 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.382361 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.382365 | controller |     - '        '
2026-03-14 09:44:27.382369 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.382374 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.382378 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.382382 | controller |     - '          exit 1'
2026-03-14 09:44:27.382386 | controller |     - '        fi'
2026-03-14 09:44:27.382390 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.382394 | controller |     - '        '
2026-03-14 09:44:27.382399 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.382403 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.382407 | controller |       | base64 -d)'
2026-03-14 09:44:27.382411 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.382415 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.382420 | controller |     - '          exit 1'
2026-03-14 09:44:27.382424 | controller |     - '        fi'
2026-03-14 09:44:27.382428 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.382432 | controller |     - '        '
2026-03-14 09:44:27.382437 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.382441 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.382445 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.382449 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.382453 | controller |       --timeout=300s'
2026-03-14 09:44:27.382458 | controller |     - '        '
2026-03-14 09:44:27.382462 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.382466 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.382470 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.382474 | controller |     - '        '
2026-03-14 09:44:27.382479 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.382483 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.382487 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.382491 | controller |     - '        '
2026-03-14 09:44:27.382495 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.382500 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.382504 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.382508 | controller |       available"'
2026-03-14 09:44:27.382512 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.382516 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.382521 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.382525 | controller |     - '          exit 1'
2026-03-14 09:44:27.382529 | controller |     - '        fi'
2026-03-14 09:44:27.382533 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.382539 | controller |     - '        '
2026-03-14 09:44:27.382543 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.382548 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.382552 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.382556 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.382562 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.382566 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.382570 | controller |       -- \'
2026-03-14 09:44:27.382574 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.382584 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.382589 | controller |     - '    logger.go:42: 09:34:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.382593 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.382597 | controller |     - '    logger.go:42: 09:34:56 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.382602 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.382606 | controller |       found'
2026-03-14 09:44:27.382610 | controller |     - '    logger.go:42: 09:34:57 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.382614 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.382619 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.382625 | controller |     - '        '
2026-03-14 09:44:27.382631 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.382637 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.382645 | controller |     - '        '
2026-03-14 09:44:27.382651 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.382655 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.382659 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.382664 | controller |     - '          exit 1'
2026-03-14 09:44:27.382676 | controller |     - '        fi'
2026-03-14 09:44:27.382681 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.382686 | controller |     - '        '
2026-03-14 09:44:27.382690 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.382694 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.382698 | controller |       | base64 -d)'
2026-03-14 09:44:27.382702 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.382706 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.382711 | controller |     - '          exit 1'
2026-03-14 09:44:27.382715 | controller |     - '        fi'
2026-03-14 09:44:27.382719 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.382723 | controller |     - '        '
2026-03-14 09:44:27.382727 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.382732 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.382736 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.382740 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.382744 | controller |       --timeout=300s'
2026-03-14 09:44:27.382748 | controller |     - '        '
2026-03-14 09:44:27.382753 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.382759 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.382763 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.382767 | controller |     - '        '
2026-03-14 09:44:27.382771 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.382775 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.382780 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.382784 | controller |     - '        '
2026-03-14 09:44:27.382788 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.382792 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.382796 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.382801 | controller |       available"'
2026-03-14 09:44:27.382805 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.382809 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.382813 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.382817 | controller |     - '          exit 1'
2026-03-14 09:44:27.382821 | controller |     - '        fi'
2026-03-14 09:44:27.382826 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.382830 | controller |     - '        '
2026-03-14 09:44:27.382834 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.382838 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.382842 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.382846 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.382851 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.382855 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.382859 | controller |       -- \'
2026-03-14 09:44:27.382863 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.382867 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.382877 | controller |     - '    logger.go:42: 09:34:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.382882 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.382886 | controller |     - '    logger.go:42: 09:34:57 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.382890 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.382895 | controller |       found'
2026-03-14 09:44:27.382901 | controller |     - '    logger.go:42: 09:34:58 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.382905 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.382909 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.382913 | controller |     - '        '
2026-03-14 09:44:27.382918 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.382922 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.382926 | controller |     - '        '
2026-03-14 09:44:27.382930 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.382934 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.382941 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.382945 | controller |     - '          exit 1'
2026-03-14 09:44:27.382949 | controller |     - '        fi'
2026-03-14 09:44:27.382953 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.382957 | controller |     - '        '
2026-03-14 09:44:27.382962 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.382966 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.382970 | controller |       | base64 -d)'
2026-03-14 09:44:27.382974 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.382979 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.382983 | controller |     - '          exit 1'
2026-03-14 09:44:27.382987 | controller |     - '        fi'
2026-03-14 09:44:27.382991 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.382996 | controller |     - '        '
2026-03-14 09:44:27.383000 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.383004 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.383008 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.383013 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.383017 | controller |       --timeout=300s'
2026-03-14 09:44:27.383021 | controller |     - '        '
2026-03-14 09:44:27.383025 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.383029 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.383034 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.383038 | controller |     - '        '
2026-03-14 09:44:27.383044 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.383048 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.383053 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.383057 | controller |     - '        '
2026-03-14 09:44:27.383061 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.383065 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.383070 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.383074 | controller |       available"'
2026-03-14 09:44:27.383078 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.383082 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.383086 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.383091 | controller |     - '          exit 1'
2026-03-14 09:44:27.383095 | controller |     - '        fi'
2026-03-14 09:44:27.383099 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.383103 | controller |     - '        '
2026-03-14 09:44:27.383108 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.383112 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.383116 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.383120 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.383124 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.383131 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.383135 | controller |       -- \'
2026-03-14 09:44:27.383139 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.383143 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.383153 | controller |     - '    logger.go:42: 09:34:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.383158 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.383162 | controller |     - '    logger.go:42: 09:34:58 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.383166 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.383170 | controller |       found'
2026-03-14 09:44:27.383174 | controller |     - '    logger.go:42: 09:34:59 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.383179 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.383183 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.383187 | controller |     - '        '
2026-03-14 09:44:27.383191 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.383195 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.383200 | controller |     - '        '
2026-03-14 09:44:27.383204 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.383208 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.383212 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.383216 | controller |     - '          exit 1'
2026-03-14 09:44:27.383220 | controller |     - '        fi'
2026-03-14 09:44:27.383225 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.383229 | controller |     - '        '
2026-03-14 09:44:27.383233 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.383237 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.383241 | controller |       | base64 -d)'
2026-03-14 09:44:27.383246 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.383250 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.383254 | controller |     - '          exit 1'
2026-03-14 09:44:27.383258 | controller |     - '        fi'
2026-03-14 09:44:27.383263 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.383267 | controller |     - '        '
2026-03-14 09:44:27.383271 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.383275 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.383279 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.383283 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.383299 | controller |       --timeout=300s'
2026-03-14 09:44:27.383307 | controller |     - '        '
2026-03-14 09:44:27.383311 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.383316 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.383320 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.383324 | controller |     - '        '
2026-03-14 09:44:27.383328 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.383333 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.383339 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.383344 | controller |     - '        '
2026-03-14 09:44:27.383348 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.383354 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.383360 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.383366 | controller |       available"'
2026-03-14 09:44:27.383372 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.383377 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.383383 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.383388 | controller |     - '          exit 1'
2026-03-14 09:44:27.383393 | controller |     - '        fi'
2026-03-14 09:44:27.383399 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.383405 | controller |     - '        '
2026-03-14 09:44:27.383410 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.383414 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.383418 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.383422 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.383427 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.383431 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.383435 | controller |       -- \'
2026-03-14 09:44:27.383439 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.383444 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.383450 | controller |     - '    logger.go:42: 09:34:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.383463 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.383470 | controller |     - '    logger.go:42: 09:34:59 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.383476 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.383482 | controller |       found'
2026-03-14 09:44:27.383487 | controller |     - '    logger.go:42: 09:35:00 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.383493 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.383499 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.383504 | controller |     - '        '
2026-03-14 09:44:27.383515 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.383521 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.383527 | controller |     - '        '
2026-03-14 09:44:27.383533 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.383539 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.383544 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.383550 | controller |     - '          exit 1'
2026-03-14 09:44:27.383556 | controller |     - '        fi'
2026-03-14 09:44:27.383562 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.383568 | controller |     - '        '
2026-03-14 09:44:27.383574 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.383579 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.383585 | controller |       | base64 -d)'
2026-03-14 09:44:27.383590 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.383594 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.383598 | controller |     - '          exit 1'
2026-03-14 09:44:27.383603 | controller |     - '        fi'
2026-03-14 09:44:27.383609 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.383615 | controller |     - '        '
2026-03-14 09:44:27.383621 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.383627 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.383633 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.383638 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.383644 | controller |       --timeout=300s'
2026-03-14 09:44:27.383650 | controller |     - '        '
2026-03-14 09:44:27.383656 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.383661 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.383667 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.383672 | controller |     - '        '
2026-03-14 09:44:27.383678 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.383687 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.383694 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.383700 | controller |     - '        '
2026-03-14 09:44:27.383706 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.383711 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.383717 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.383723 | controller |       available"'
2026-03-14 09:44:27.383729 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.383735 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.383741 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.383747 | controller |     - '          exit 1'
2026-03-14 09:44:27.383756 | controller |     - '        fi'
2026-03-14 09:44:27.383762 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.383768 | controller |     - '        '
2026-03-14 09:44:27.383774 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.383779 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.383785 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.383791 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.383797 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.383803 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.383809 | controller |       -- \'
2026-03-14 09:44:27.383814 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.383820 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.383826 | controller |     - '    logger.go:42: 09:35:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.383836 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.383852 | controller |     - '    logger.go:42: 09:35:00 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.383859 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.383865 | controller |       found'
2026-03-14 09:44:27.383871 | controller |     - '    logger.go:42: 09:35:01 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.383877 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.383882 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.383888 | controller |     - '        '
2026-03-14 09:44:27.383893 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.383900 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.383905 | controller |     - '        '
2026-03-14 09:44:27.383911 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.383917 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.383923 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.383929 | controller |     - '          exit 1'
2026-03-14 09:44:27.383933 | controller |     - '        fi'
2026-03-14 09:44:27.383937 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.383941 | controller |     - '        '
2026-03-14 09:44:27.383947 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.383953 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.383959 | controller |       | base64 -d)'
2026-03-14 09:44:27.383965 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.383971 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.383977 | controller |     - '          exit 1'
2026-03-14 09:44:27.383983 | controller |     - '        fi'
2026-03-14 09:44:27.383988 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.383994 | controller |     - '        '
2026-03-14 09:44:27.384000 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.384006 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.384012 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.384017 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.384022 | controller |       --timeout=300s'
2026-03-14 09:44:27.384027 | controller |     - '        '
2026-03-14 09:44:27.384032 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.384038 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.384043 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.384048 | controller |     - '        '
2026-03-14 09:44:27.384053 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.384058 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.384063 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.384068 | controller |     - '        '
2026-03-14 09:44:27.384073 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.384078 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.384083 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.384187 | controller |       available"'
2026-03-14 09:44:27.384192 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.384196 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.384200 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.384204 | controller |     - '          exit 1'
2026-03-14 09:44:27.384209 | controller |     - '        fi'
2026-03-14 09:44:27.384213 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.384217 | controller |     - '        '
2026-03-14 09:44:27.384221 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.384225 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.384230 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.384234 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.384238 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.384242 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.384246 | controller |       -- \'
2026-03-14 09:44:27.384251 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.384255 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.384259 | controller |     - '    logger.go:42: 09:35:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.384263 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.384275 | controller |     - '    logger.go:42: 09:35:02 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.384280 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.384284 | controller |       found'
2026-03-14 09:44:27.384303 | controller |     - '    logger.go:42: 09:35:03 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.384311 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.384315 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.384319 | controller |     - '        '
2026-03-14 09:44:27.384323 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.384331 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.384335 | controller |     - '        '
2026-03-14 09:44:27.384339 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.384343 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.384348 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.384352 | controller |     - '          exit 1'
2026-03-14 09:44:27.384356 | controller |     - '        fi'
2026-03-14 09:44:27.384360 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.384364 | controller |     - '        '
2026-03-14 09:44:27.384369 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.384373 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.384377 | controller |       | base64 -d)'
2026-03-14 09:44:27.384381 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.384385 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.384389 | controller |     - '          exit 1'
2026-03-14 09:44:27.384394 | controller |     - '        fi'
2026-03-14 09:44:27.384398 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.384402 | controller |     - '        '
2026-03-14 09:44:27.384406 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.384412 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.384417 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.384421 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.384425 | controller |       --timeout=300s'
2026-03-14 09:44:27.384429 | controller |     - '        '
2026-03-14 09:44:27.384433 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.384437 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.384442 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.384446 | controller |     - '        '
2026-03-14 09:44:27.384450 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.384454 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.384458 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.384462 | controller |     - '        '
2026-03-14 09:44:27.384467 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.384471 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.384475 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.384479 | controller |       available"'
2026-03-14 09:44:27.384483 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.384487 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.384492 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.384496 | controller |     - '          exit 1'
2026-03-14 09:44:27.384500 | controller |     - '        fi'
2026-03-14 09:44:27.384504 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.384508 | controller |     - '        '
2026-03-14 09:44:27.384512 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.384517 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.384521 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.384525 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.384529 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.384533 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.384537 | controller |       -- \'
2026-03-14 09:44:27.384542 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.384546 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.384550 | controller |     - '    logger.go:42: 09:35:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.384554 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.384558 | controller |     - '    logger.go:42: 09:35:04 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.384569 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.384573 | controller |       found'
2026-03-14 09:44:27.384578 | controller |     - '    logger.go:42: 09:35:05 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.384582 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.384588 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.384592 | controller |     - '        '
2026-03-14 09:44:27.384596 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.384600 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.384605 | controller |     - '        '
2026-03-14 09:44:27.384609 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.384613 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.384619 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.384625 | controller |     - '          exit 1'
2026-03-14 09:44:27.384631 | controller |     - '        fi'
2026-03-14 09:44:27.384637 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.384643 | controller |     - '        '
2026-03-14 09:44:27.384649 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.384654 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.384658 | controller |       | base64 -d)'
2026-03-14 09:44:27.384662 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.384669 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.384673 | controller |     - '          exit 1'
2026-03-14 09:44:27.384677 | controller |     - '        fi'
2026-03-14 09:44:27.384682 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.384686 | controller |     - '        '
2026-03-14 09:44:27.384690 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.384694 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.384698 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.384703 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.384707 | controller |       --timeout=300s'
2026-03-14 09:44:27.384711 | controller |     - '        '
2026-03-14 09:44:27.384716 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.384720 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.384724 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.384729 | controller |     - '        '
2026-03-14 09:44:27.384733 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.384737 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.384741 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.384745 | controller |     - '        '
2026-03-14 09:44:27.384750 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.384754 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.384758 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.384762 | controller |       available"'
2026-03-14 09:44:27.384766 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.384770 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.384774 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.384779 | controller |     - '          exit 1'
2026-03-14 09:44:27.384783 | controller |     - '        fi'
2026-03-14 09:44:27.384787 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.384791 | controller |     - '        '
2026-03-14 09:44:27.384800 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.384804 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.384808 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.384812 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.384817 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.384821 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.384827 | controller |       -- \'
2026-03-14 09:44:27.384831 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.384835 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.384839 | controller |     - '    logger.go:42: 09:35:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.384844 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.384848 | controller |     - '    logger.go:42: 09:35:05 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.384852 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.384863 | controller |       found'
2026-03-14 09:44:27.384868 | controller |     - '    logger.go:42: 09:35:06 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.384872 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.384876 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.384881 | controller |     - '        '
2026-03-14 09:44:27.384885 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.384889 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.384893 | controller |     - '        '
2026-03-14 09:44:27.384907 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.384912 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.384916 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.384920 | controller |     - '          exit 1'
2026-03-14 09:44:27.384925 | controller |     - '        fi'
2026-03-14 09:44:27.384929 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.384933 | controller |     - '        '
2026-03-14 09:44:27.384937 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.384942 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.384948 | controller |       | base64 -d)'
2026-03-14 09:44:27.384954 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.384960 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.384966 | controller |     - '          exit 1'
2026-03-14 09:44:27.384972 | controller |     - '        fi'
2026-03-14 09:44:27.384978 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.384984 | controller |     - '        '
2026-03-14 09:44:27.384989 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.384995 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.385001 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.385007 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.385013 | controller |       --timeout=300s'
2026-03-14 09:44:27.385024 | controller |     - '        '
2026-03-14 09:44:27.385029 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.385035 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.385044 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.385050 | controller |     - '        '
2026-03-14 09:44:27.385056 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.385061 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.385067 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.385073 | controller |     - '        '
2026-03-14 09:44:27.385079 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.385085 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.385090 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.385094 | controller |       available"'
2026-03-14 09:44:27.385098 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.385103 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.385107 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.385111 | controller |     - '          exit 1'
2026-03-14 09:44:27.385119 | controller |     - '        fi'
2026-03-14 09:44:27.385123 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.385128 | controller |     - '        '
2026-03-14 09:44:27.385132 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.385136 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.385140 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.385144 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.385149 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.385153 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.385157 | controller |       -- \'
2026-03-14 09:44:27.385161 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.385165 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.385170 | controller |     - '    logger.go:42: 09:35:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.385174 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.385178 | controller |     - '    logger.go:42: 09:35:06 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.385182 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.385186 | controller |       found'
2026-03-14 09:44:27.385198 | controller |     - '    logger.go:42: 09:35:07 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.385203 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.385208 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.385212 | controller |     - '        '
2026-03-14 09:44:27.385216 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.385220 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.385224 | controller |     - '        '
2026-03-14 09:44:27.385229 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.385233 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.385237 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.385243 | controller |     - '          exit 1'
2026-03-14 09:44:27.385248 | controller |     - '        fi'
2026-03-14 09:44:27.385252 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.385256 | controller |     - '        '
2026-03-14 09:44:27.385260 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.385264 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.385269 | controller |       | base64 -d)'
2026-03-14 09:44:27.385273 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.385277 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.385281 | controller |     - '          exit 1'
2026-03-14 09:44:27.385285 | controller |     - '        fi'
2026-03-14 09:44:27.385307 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.385314 | controller |     - '        '
2026-03-14 09:44:27.385318 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.385323 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.385327 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.385331 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.385335 | controller |       --timeout=300s'
2026-03-14 09:44:27.385339 | controller |     - '        '
2026-03-14 09:44:27.385343 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.385348 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.385352 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.385356 | controller |     - '        '
2026-03-14 09:44:27.385360 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.385364 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.385369 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.385373 | controller |     - '        '
2026-03-14 09:44:27.385377 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.385381 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.385385 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.385389 | controller |       available"'
2026-03-14 09:44:27.385394 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.385398 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.385402 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.385406 | controller |     - '          exit 1'
2026-03-14 09:44:27.385410 | controller |     - '        fi'
2026-03-14 09:44:27.385415 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.385419 | controller |     - '        '
2026-03-14 09:44:27.385423 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.385427 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.385431 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.385435 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.385440 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.385446 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.385450 | controller |       -- \'
2026-03-14 09:44:27.385454 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.385459 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.385463 | controller |     - '    logger.go:42: 09:35:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.385467 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.385471 | controller |     - '    logger.go:42: 09:35:08 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.385475 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.385479 | controller |       found'
2026-03-14 09:44:27.385484 | controller |     - '    logger.go:42: 09:35:09 | appcred-tests/2-deploy-appcred | running command:
2026-03-14 09:44:27.385493 | controller |       [sh -c set -euo pipefail'
2026-03-14 09:44:27.385498 | controller |     - '        NS="${NAMESPACE}"'
2026-03-14 09:44:27.385502 | controller |     - '        '
2026-03-14 09:44:27.385507 | controller |     - '        echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."'
2026-03-14 09:44:27.385511 | controller |     - '        oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s'
2026-03-14 09:44:27.385515 | controller |     - '        '
2026-03-14 09:44:27.385519 | controller |     - '        ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')'
2026-03-14 09:44:27.385523 | controller |     - '        if [ -z "${ac_id}" ]; then'
2026-03-14 09:44:27.385528 | controller |     - '          echo "ERROR: appcred/ac-watcher.status.acID is empty"'
2026-03-14 09:44:27.385532 | controller |     - '          exit 1'
2026-03-14 09:44:27.385536 | controller |     - '        fi'
2026-03-14 09:44:27.385540 | controller |     - '        echo "ac-watcher.status.acID=${ac_id}"'
2026-03-14 09:44:27.385547 | controller |     - '        '
2026-03-14 09:44:27.385551 | controller |     - '        oc get -n "${NS}" secret/ac-watcher-secret >/dev/null'
2026-03-14 09:44:27.385555 | controller |     - '        secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}''
2026-03-14 09:44:27.385560 | controller |       | base64 -d)'
2026-03-14 09:44:27.385564 | controller |     - '        if [ "${secret_ac_id}" != "${ac_id}" ]; then'
2026-03-14 09:44:27.385568 | controller |     - '          echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"'
2026-03-14 09:44:27.385572 | controller |     - '          exit 1'
2026-03-14 09:44:27.385576 | controller |     - '        fi'
2026-03-14 09:44:27.385581 | controller |     - '        echo "✓ Secret AC_ID matches appcred.status.acID"'
2026-03-14 09:44:27.385585 | controller |     - '        '
2026-03-14 09:44:27.385589 | controller |     - '        echo "Waiting for watcher rollouts..."'
2026-03-14 09:44:27.385593 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s'
2026-03-14 09:44:27.385597 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s'
2026-03-14 09:44:27.385602 | controller |     - '        oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine
2026-03-14 09:44:27.385606 | controller |       --timeout=300s'
2026-03-14 09:44:27.385610 | controller |     - '        '
2026-03-14 09:44:27.385614 | controller |     - '        old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')'
2026-03-14 09:44:27.385618 | controller |     - '        old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')'
2026-03-14 09:44:27.385622 | controller |     - '        old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')'
2026-03-14 09:44:27.385627 | controller |     - '        '
2026-03-14 09:44:27.385632 | controller |     - '        new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.385638 | controller |     - '        new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.385646 | controller |     - '        new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')'
2026-03-14 09:44:27.385652 | controller |     - '        '
2026-03-14 09:44:27.385657 | controller |     - '        if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" =
2026-03-14 09:44:27.385662 | controller |       "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then'
2026-03-14 09:44:27.385666 | controller |     - '          echo "ERROR: expected watcher pods to restart after appcred secret became
2026-03-14 09:44:27.385670 | controller |       available"'
2026-03-14 09:44:27.385674 | controller |     - '          echo "  api: ${old_api_uid} -> ${new_api_uid}"'
2026-03-14 09:44:27.385679 | controller |     - '          echo "  applier: ${old_applier_uid} -> ${new_applier_uid}"'
2026-03-14 09:44:27.385683 | controller |     - '          echo "  decision-engine: ${old_de_uid} -> ${new_de_uid}"'
2026-03-14 09:44:27.385687 | controller |     - '          exit 1'
2026-03-14 09:44:27.385691 | controller |     - '        fi'
2026-03-14 09:44:27.385695 | controller |     - '        echo "✓ watcher pods restarted after appcred secret became available"'
2026-03-14 09:44:27.385699 | controller |     - '        '
2026-03-14 09:44:27.385704 | controller |     - '        echo "Checking watcher config contains application_credential_id..."'
2026-03-14 09:44:27.385708 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \'
2026-03-14 09:44:27.385712 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.385716 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \'
2026-03-14 09:44:27.385720 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.385724 | controller |     - '        oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine
2026-03-14 09:44:27.385729 | controller |       -- \'
2026-03-14 09:44:27.385733 | controller |     - '          bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"'
2026-03-14 09:44:27.385737 | controller |     - '        echo "✓ watcher config contains expected application_credential_id"]'
2026-03-14 09:44:27.385741 | controller |     - '    logger.go:42: 09:35:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential
2026-03-14 09:44:27.385745 | controller |       ac-watcher to be Ready...'
2026-03-14 09:44:27.385750 | controller |     - '    logger.go:42: 09:35:09 | appcred-tests/2-deploy-appcred | Error from server
2026-03-14 09:44:27.385754 | controller |       (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not
2026-03-14 09:44:27.385758 | controller |       found'
2026-03-14 09:44:27.385762 | controller |     - '    logger.go:42: 09:35:10 | appcred-tests/2-deploy-appcred | test step failed
2026-03-14 09:44:27.385766 | controller |       2-deploy-appcred'
2026-03-14 09:44:27.385776 | controller |     - '    case.go:396: failed in step 2-deploy-appcred'
2026-03-14 09:44:27.385781 | controller |     - '    case.go:398: command "NS=\"${NAMESPACE}\"\\n echo \"Waiting for KeystoneApplicationCredential..."
2026-03-14 09:44:27.385785 | controller |       failed, exit status 1'
2026-03-14 09:44:27.385789 | controller |     - '    logger.go:42: 09:35:10 | appcred-tests | skipping kubernetes event logging'
2026-03-14 09:44:27.385793 | controller |     - === CONT  kuttl/harness/watcher-notification
2026-03-14 09:44:27.385798 | controller |     - '    logger.go:42: 09:35:10 | watcher-notification | Skipping creation of user-supplied
2026-03-14 09:44:27.385802 | controller |       namespace: watcher-kuttl-default'
2026-03-14 09:44:27.385809 | controller |     - '    logger.go:42: 09:35:10 | watcher-notification/0-cleanup-watcher | starting
2026-03-14 09:44:27.385814 | controller |       test step 0-cleanup-watcher'
2026-03-14 09:44:27.385818 | controller |     - '    logger.go:42: 09:35:10 | watcher-notification/0-cleanup-watcher | test step
2026-03-14 09:44:27.385822 | controller |       completed 0-cleanup-watcher'
2026-03-14 09:44:27.385826 | controller |     - '    logger.go:42: 09:35:10 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.385831 | controller |       starting test step 1-deploy-with-notification'
2026-03-14 09:44:27.385837 | controller |     - '    logger.go:42: 09:35:10 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.385842 | controller |       Watcher:watcher-kuttl-default/watcher-kuttl created'
2026-03-14 09:44:27.385846 | controller |     - '    logger.go:42: 09:35:10 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.385850 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.385854 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.385858 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.385863 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.385867 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.385871 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.385875 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.385879 | controller |       == 1 ]'
2026-03-14 09:44:27.385883 | controller |     - '        else'
2026-03-14 09:44:27.385888 | controller |     - '          exit 1'
2026-03-14 09:44:27.385892 | controller |     - '        fi'
2026-03-14 09:44:27.385896 | controller |     - '        ]'
2026-03-14 09:44:27.385900 | controller |     - '    logger.go:42: 09:35:10 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.385905 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.385909 | controller |     - '    logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.385913 | controller |       Now using project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.385917 | controller |     - '    logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.385921 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.385925 | controller |     - '    logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.385930 | controller |       ++ head -1'
2026-03-14 09:44:27.385934 | controller |     - '    logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.385938 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.385942 | controller |     - '    logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.385946 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.385951 | controller |     - '    logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.385955 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.385959 | controller |     - '    logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.385963 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:44:27.385967 | controller |     - '    logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.385973 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.385978 | controller |     - '    logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.385983 | controller |       error: Internal error occurred: error executing command in container: container
2026-03-14 09:44:27.385988 | controller |       is not created or running'
2026-03-14 09:44:27.385992 | controller |     - '    logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.385996 | controller |       ++ echo'
2026-03-14 09:44:27.386000 | controller |     - '    logger.go:42: 09:35:11 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386004 | controller |       + ''['' 0 == 1 '']'''
2026-03-14 09:44:27.386009 | controller |     - '    logger.go:42: 09:35:12 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386013 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.386017 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.386021 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.386028 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.386032 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.386036 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.386041 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.386045 | controller |       == 1 ]'
2026-03-14 09:44:27.386049 | controller |     - '        else'
2026-03-14 09:44:27.386053 | controller |     - '          exit 1'
2026-03-14 09:44:27.386057 | controller |     - '        fi'
2026-03-14 09:44:27.386062 | controller |     - '        ]'
2026-03-14 09:44:27.386071 | controller |     - '    logger.go:42: 09:35:12 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386076 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.386080 | controller |     - '    logger.go:42: 09:35:13 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386084 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.386088 | controller |     - '    logger.go:42: 09:35:13 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386119 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.386124 | controller |     - '    logger.go:42: 09:35:13 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386129 | controller |       ++ head -1'
2026-03-14 09:44:27.386134 | controller |     - '    logger.go:42: 09:35:13 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386139 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.386144 | controller |     - '    logger.go:42: 09:35:13 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386149 | controller |       + APIPOD='
2026-03-14 09:44:27.386154 | controller |     - '    logger.go:42: 09:35:14 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386159 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.386164 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.386169 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.386174 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.386179 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.386184 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.386188 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.386193 | controller |       == 1 ]'
2026-03-14 09:44:27.386198 | controller |     - '        else'
2026-03-14 09:44:27.386203 | controller |     - '          exit 1'
2026-03-14 09:44:27.386208 | controller |     - '        fi'
2026-03-14 09:44:27.386213 | controller |     - '        ]'
2026-03-14 09:44:27.386218 | controller |     - '    logger.go:42: 09:35:14 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386223 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.386228 | controller |     - '    logger.go:42: 09:35:14 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386233 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.386239 | controller |     - '    logger.go:42: 09:35:14 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386244 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.386250 | controller |     - '    logger.go:42: 09:35:14 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386256 | controller |       ++ head -1'
2026-03-14 09:44:27.386262 | controller |     - '    logger.go:42: 09:35:14 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386267 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.386276 | controller |     - '    logger.go:42: 09:35:14 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386282 | controller |       + APIPOD='
2026-03-14 09:44:27.386306 | controller |     - '    logger.go:42: 09:35:15 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386315 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.386319 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.386323 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.386328 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.386332 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.386336 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.386340 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.386345 | controller |       == 1 ]'
2026-03-14 09:44:27.386349 | controller |     - '        else'
2026-03-14 09:44:27.386354 | controller |     - '          exit 1'
2026-03-14 09:44:27.386360 | controller |     - '        fi'
2026-03-14 09:44:27.386366 | controller |     - '        ]'
2026-03-14 09:44:27.386372 | controller |     - '    logger.go:42: 09:35:15 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386378 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.386384 | controller |     - '    logger.go:42: 09:35:15 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386389 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.386396 | controller |     - '    logger.go:42: 09:35:15 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386401 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.386407 | controller |     - '    logger.go:42: 09:35:15 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386413 | controller |       ++ head -1'
2026-03-14 09:44:27.386419 | controller |     - '    logger.go:42: 09:35:15 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386425 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.386431 | controller |     - '    logger.go:42: 09:35:15 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386437 | controller |       + APIPOD='
2026-03-14 09:44:27.386442 | controller |     - '    logger.go:42: 09:35:16 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386449 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.386455 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.386466 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.386482 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.386489 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.386495 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.386501 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.386507 | controller |       == 1 ]'
2026-03-14 09:44:27.386513 | controller |     - '        else'
2026-03-14 09:44:27.386519 | controller |     - '          exit 1'
2026-03-14 09:44:27.386524 | controller |     - '        fi'
2026-03-14 09:44:27.386528 | controller |     - '        ]'
2026-03-14 09:44:27.386532 | controller |     - '    logger.go:42: 09:35:16 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386537 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.386541 | controller |     - '    logger.go:42: 09:35:17 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386545 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.386549 | controller |     - '    logger.go:42: 09:35:17 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386553 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.386557 | controller |     - '    logger.go:42: 09:35:17 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386562 | controller |       ++ head -1'
2026-03-14 09:44:27.386568 | controller |     - '    logger.go:42: 09:35:17 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386573 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.386577 | controller |     - '    logger.go:42: 09:35:17 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386581 | controller |       + APIPOD='
2026-03-14 09:44:27.386585 | controller |     - '    logger.go:42: 09:35:18 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386590 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.386594 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.386599 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.386605 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.386611 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.386617 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.386623 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.386628 | controller |       == 1 ]'
2026-03-14 09:44:27.386634 | controller |     - '        else'
2026-03-14 09:44:27.386640 | controller |     - '          exit 1'
2026-03-14 09:44:27.386645 | controller |     - '        fi'
2026-03-14 09:44:27.386651 | controller |     - '        ]'
2026-03-14 09:44:27.386657 | controller |     - '    logger.go:42: 09:35:18 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386663 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.386669 | controller |     - '    logger.go:42: 09:35:18 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386674 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.386678 | controller |     - '    logger.go:42: 09:35:18 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386682 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.386687 | controller |     - '    logger.go:42: 09:35:18 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386691 | controller |       ++ head -1'
2026-03-14 09:44:27.386695 | controller |     - '    logger.go:42: 09:35:18 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386699 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.386703 | controller |     - '    logger.go:42: 09:35:18 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386708 | controller |       + APIPOD='
2026-03-14 09:44:27.386712 | controller |     - '    logger.go:42: 09:35:19 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386716 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.386720 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.386724 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.386728 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.386733 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.386737 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.386741 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.386745 | controller |       == 1 ]'
2026-03-14 09:44:27.386749 | controller |     - '        else'
2026-03-14 09:44:27.386753 | controller |     - '          exit 1'
2026-03-14 09:44:27.386758 | controller |     - '        fi'
2026-03-14 09:44:27.386762 | controller |     - '        ]'
2026-03-14 09:44:27.386766 | controller |     - '    logger.go:42: 09:35:19 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386770 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.386775 | controller |     - '    logger.go:42: 09:35:19 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386779 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.386783 | controller |     - '    logger.go:42: 09:35:19 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386790 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.386794 | controller |     - '    logger.go:42: 09:35:19 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386798 | controller |       ++ head -1'
2026-03-14 09:44:27.386802 | controller |     - '    logger.go:42: 09:35:19 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386806 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.386818 | controller |     - '    logger.go:42: 09:35:19 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386823 | controller |       + APIPOD='
2026-03-14 09:44:27.386827 | controller |     - '    logger.go:42: 09:35:20 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386832 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.386836 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.386840 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.386844 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.386848 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.386852 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.386857 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.386861 | controller |       == 1 ]'
2026-03-14 09:44:27.386865 | controller |     - '        else'
2026-03-14 09:44:27.386869 | controller |     - '          exit 1'
2026-03-14 09:44:27.386873 | controller |     - '        fi'
2026-03-14 09:44:27.386878 | controller |     - '        ]'
2026-03-14 09:44:27.386882 | controller |     - '    logger.go:42: 09:35:20 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386886 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.386890 | controller |     - '    logger.go:42: 09:35:20 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386894 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.386901 | controller |     - '    logger.go:42: 09:35:20 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386905 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.386910 | controller |     - '    logger.go:42: 09:35:20 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386914 | controller |       ++ head -1'
2026-03-14 09:44:27.386918 | controller |     - '    logger.go:42: 09:35:20 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386922 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.386926 | controller |     - '    logger.go:42: 09:35:21 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386931 | controller |       + APIPOD='
2026-03-14 09:44:27.386937 | controller |     - '    logger.go:42: 09:35:22 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.386942 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.386948 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.386953 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.386958 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.386965 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.386970 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.386977 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.386982 | controller |       == 1 ]'
2026-03-14 09:44:27.386989 | controller |     - '        else'
2026-03-14 09:44:27.386995 | controller |     - '          exit 1'
2026-03-14 09:44:27.387001 | controller |     - '        fi'
2026-03-14 09:44:27.387007 | controller |     - '        ]'
2026-03-14 09:44:27.387188 | controller |     - '    logger.go:42: 09:35:22 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387194 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.387204 | controller |     - '    logger.go:42: 09:35:22 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387209 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.387213 | controller |     - '    logger.go:42: 09:35:22 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387217 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.387222 | controller |     - '    logger.go:42: 09:35:22 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387226 | controller |       ++ head -1'
2026-03-14 09:44:27.387230 | controller |     - '    logger.go:42: 09:35:22 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387234 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.387238 | controller |     - '    logger.go:42: 09:35:22 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387242 | controller |       + APIPOD='
2026-03-14 09:44:27.387247 | controller |     - '    logger.go:42: 09:35:23 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387251 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.387255 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.387259 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.387263 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.387267 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.387272 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.387276 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.387280 | controller |       == 1 ]'
2026-03-14 09:44:27.387284 | controller |     - '        else'
2026-03-14 09:44:27.387307 | controller |     - '          exit 1'
2026-03-14 09:44:27.387312 | controller |     - '        fi'
2026-03-14 09:44:27.387316 | controller |     - '        ]'
2026-03-14 09:44:27.387320 | controller |     - '    logger.go:42: 09:35:23 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387324 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.387329 | controller |     - '    logger.go:42: 09:35:23 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387333 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.387344 | controller |     - '    logger.go:42: 09:35:23 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387348 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.387353 | controller |     - '    logger.go:42: 09:35:23 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387357 | controller |       ++ head -1'
2026-03-14 09:44:27.387361 | controller |     - '    logger.go:42: 09:35:23 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387365 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.387370 | controller |     - '    logger.go:42: 09:35:23 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387374 | controller |       + APIPOD='
2026-03-14 09:44:27.387378 | controller |     - '    logger.go:42: 09:35:24 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387382 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.387386 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.387390 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.387397 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.387402 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.387407 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.387412 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.387416 | controller |       == 1 ]'
2026-03-14 09:44:27.387422 | controller |     - '        else'
2026-03-14 09:44:27.387426 | controller |     - '          exit 1'
2026-03-14 09:44:27.387430 | controller |     - '        fi'
2026-03-14 09:44:27.387435 | controller |     - '        ]'
2026-03-14 09:44:27.387439 | controller |     - '    logger.go:42: 09:35:24 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387443 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.387447 | controller |     - '    logger.go:42: 09:35:24 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387451 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.387455 | controller |     - '    logger.go:42: 09:35:24 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387460 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.387464 | controller |     - '    logger.go:42: 09:35:24 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387468 | controller |       ++ head -1'
2026-03-14 09:44:27.387472 | controller |     - '    logger.go:42: 09:35:24 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387476 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.387480 | controller |     - '    logger.go:42: 09:35:24 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387485 | controller |       + APIPOD='
2026-03-14 09:44:27.387490 | controller |     - '    logger.go:42: 09:35:25 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387494 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.387499 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.387503 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.387507 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.387511 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.387515 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.387519 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.387524 | controller |       == 1 ]'
2026-03-14 09:44:27.387528 | controller |     - '        else'
2026-03-14 09:44:27.387532 | controller |     - '          exit 1'
2026-03-14 09:44:27.387536 | controller |     - '        fi'
2026-03-14 09:44:27.387540 | controller |     - '        ]'
2026-03-14 09:44:27.387545 | controller |     - '    logger.go:42: 09:35:25 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387549 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.387553 | controller |     - '    logger.go:42: 09:35:26 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387557 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.387561 | controller |     - '    logger.go:42: 09:35:26 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387565 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.387570 | controller |     - '    logger.go:42: 09:35:26 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387574 | controller |       ++ head -1'
2026-03-14 09:44:27.387578 | controller |     - '    logger.go:42: 09:35:26 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387582 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.387588 | controller |     - '    logger.go:42: 09:35:26 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387592 | controller |       + APIPOD='
2026-03-14 09:44:27.387597 | controller |     - '    logger.go:42: 09:35:27 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387601 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.387605 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.387609 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.387614 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.387620 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.387628 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.387634 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.387640 | controller |       == 1 ]'
2026-03-14 09:44:27.387645 | controller |     - '        else'
2026-03-14 09:44:27.387659 | controller |     - '          exit 1'
2026-03-14 09:44:27.387666 | controller |     - '        fi'
2026-03-14 09:44:27.387671 | controller |     - '        ]'
2026-03-14 09:44:27.387677 | controller |     - '    logger.go:42: 09:35:27 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387683 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.387688 | controller |     - '    logger.go:42: 09:35:27 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387692 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.387696 | controller |     - '    logger.go:42: 09:35:27 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387701 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.387705 | controller |     - '    logger.go:42: 09:35:27 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387709 | controller |       ++ head -1'
2026-03-14 09:44:27.387713 | controller |     - '    logger.go:42: 09:35:27 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387717 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.387722 | controller |     - '    logger.go:42: 09:35:27 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387726 | controller |       + APIPOD='
2026-03-14 09:44:27.387730 | controller |     - '    logger.go:42: 09:35:28 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387734 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.387738 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.387742 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.387747 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.387751 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.387755 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.387759 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.387764 | controller |       == 1 ]'
2026-03-14 09:44:27.387768 | controller |     - '        else'
2026-03-14 09:44:27.387772 | controller |     - '          exit 1'
2026-03-14 09:44:27.387776 | controller |     - '        fi'
2026-03-14 09:44:27.387780 | controller |     - '        ]'
2026-03-14 09:44:27.387784 | controller |     - '    logger.go:42: 09:35:28 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387789 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.387793 | controller |     - '    logger.go:42: 09:35:28 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387797 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.387801 | controller |     - '    logger.go:42: 09:35:28 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387805 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.387810 | controller |     - '    logger.go:42: 09:35:28 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387814 | controller |       ++ head -1'
2026-03-14 09:44:27.387818 | controller |     - '    logger.go:42: 09:35:28 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387822 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.387826 | controller |     - '    logger.go:42: 09:35:28 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387831 | controller |       + APIPOD='
2026-03-14 09:44:27.387835 | controller |     - '    logger.go:42: 09:35:29 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387839 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.387846 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.387850 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.387854 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.387858 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.387862 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.387867 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.387871 | controller |       == 1 ]'
2026-03-14 09:44:27.387875 | controller |     - '        else'
2026-03-14 09:44:27.387879 | controller |     - '          exit 1'
2026-03-14 09:44:27.387883 | controller |     - '        fi'
2026-03-14 09:44:27.387888 | controller |     - '        ]'
2026-03-14 09:44:27.387892 | controller |     - '    logger.go:42: 09:35:29 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387896 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.387900 | controller |     - '    logger.go:42: 09:35:30 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387907 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.387911 | controller |     - '    logger.go:42: 09:35:30 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387915 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.387919 | controller |     - '    logger.go:42: 09:35:30 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387924 | controller |       ++ head -1'
2026-03-14 09:44:27.387928 | controller |     - '    logger.go:42: 09:35:30 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387932 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.387936 | controller |     - '    logger.go:42: 09:35:30 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387940 | controller |       + APIPOD='
2026-03-14 09:44:27.387945 | controller |     - '    logger.go:42: 09:35:31 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.387949 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.387953 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.387963 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.387968 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.387972 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.387976 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.387981 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.387985 | controller |       == 1 ]'
2026-03-14 09:44:27.387989 | controller |     - '        else'
2026-03-14 09:44:27.387993 | controller |     - '          exit 1'
2026-03-14 09:44:27.387998 | controller |     - '        fi'
2026-03-14 09:44:27.388002 | controller |     - '        ]'
2026-03-14 09:44:27.388006 | controller |     - '    logger.go:42: 09:35:31 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388011 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.388015 | controller |     - '    logger.go:42: 09:35:31 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388019 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.388023 | controller |     - '    logger.go:42: 09:35:31 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388027 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.388032 | controller |     - '    logger.go:42: 09:35:31 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388036 | controller |       ++ head -1'
2026-03-14 09:44:27.388040 | controller |     - '    logger.go:42: 09:35:31 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388044 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.388050 | controller |     - '    logger.go:42: 09:35:31 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388055 | controller |       + APIPOD='
2026-03-14 09:44:27.388059 | controller |     - '    logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388063 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.388067 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.388072 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.388076 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.388080 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.388084 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.388088 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.388093 | controller |       == 1 ]'
2026-03-14 09:44:27.388097 | controller |     - '        else'
2026-03-14 09:44:27.388101 | controller |     - '          exit 1'
2026-03-14 09:44:27.388105 | controller |     - '        fi'
2026-03-14 09:44:27.388110 | controller |     - '        ]'
2026-03-14 09:44:27.388114 | controller |     - '    logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388118 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.388122 | controller |     - '    logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388127 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.388131 | controller |     - '    logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388135 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.388139 | controller |     - '    logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388144 | controller |       ++ head -1'
2026-03-14 09:44:27.388148 | controller |     - '    logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388152 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.388156 | controller |     - '    logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388160 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.388165 | controller |     - '    logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388169 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.388173 | controller |     - '    logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388177 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:44:27.388181 | controller |     - '    logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388185 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.388190 | controller |     - '    logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388194 | controller |       error: unable to upgrade connection: container not found ("watcher-api")'
2026-03-14 09:44:27.388198 | controller |     - '    logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388202 | controller |       ++ echo'
2026-03-14 09:44:27.388209 | controller |     - '    logger.go:42: 09:35:32 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388213 | controller |       + ''['' 0 == 1 '']'''
2026-03-14 09:44:27.388217 | controller |     - '    logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388222 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.388226 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.388232 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.388236 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.388243 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.388253 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.388258 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.388262 | controller |       == 1 ]'
2026-03-14 09:44:27.388266 | controller |     - '        else'
2026-03-14 09:44:27.388271 | controller |     - '          exit 1'
2026-03-14 09:44:27.388275 | controller |     - '        fi'
2026-03-14 09:44:27.388279 | controller |     - '        ]'
2026-03-14 09:44:27.388283 | controller |     - '    logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388301 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.388308 | controller |     - '    logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388312 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.388317 | controller |     - '    logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388321 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.388325 | controller |     - '    logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388329 | controller |       ++ head -1'
2026-03-14 09:44:27.388333 | controller |     - '    logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388338 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.388342 | controller |     - '    logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388346 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.388350 | controller |     - '    logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388354 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.388359 | controller |     - '    logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388363 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:44:27.388367 | controller |     - '    logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388371 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.388375 | controller |     - '    logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388380 | controller |       ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.388387 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.388392 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.388396 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_3add:c5d8ab0e913be98c16eb7b7c5c5a1976@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.388400 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.388405 | controller |       driver = messagingv2 transport_url = ''rabbit://default_user_YcxtplLsptl7k25cAVF:F735-UPJOup8HqlYaZyKNnsXxG0GtseU@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.388409 | controller |       ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.388414 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.388418 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:44:27.388422 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:44:27.388428 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.388432 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.388437 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.388441 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.388445 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.388449 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.388454 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.388458 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.388462 | controller |       tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:44:27.388466 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]''
2026-03-14 09:44:27.388471 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.388481 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.388486 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.388490 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.388494 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.388499 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.388503 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.388507 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.388511 | controller |     - '    logger.go:42: 09:35:34 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388516 | controller |       + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.388520 | controller |     - '    logger.go:42: 09:35:35 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388524 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.388528 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.388533 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.388537 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.388541 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.388545 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.388550 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.388554 | controller |       == 1 ]'
2026-03-14 09:44:27.388558 | controller |     - '        else'
2026-03-14 09:44:27.388563 | controller |     - '          exit 1'
2026-03-14 09:44:27.388567 | controller |     - '        fi'
2026-03-14 09:44:27.388571 | controller |     - '        ]'
2026-03-14 09:44:27.388575 | controller |     - '    logger.go:42: 09:35:35 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388580 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.388584 | controller |     - '    logger.go:42: 09:35:35 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388588 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.388592 | controller |     - '    logger.go:42: 09:35:35 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388598 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.388604 | controller |     - '    logger.go:42: 09:35:35 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388609 | controller |       ++ head -1'
2026-03-14 09:44:27.388613 | controller |     - '    logger.go:42: 09:35:35 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388617 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.388621 | controller |     - '    logger.go:42: 09:35:35 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388626 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.388630 | controller |     - '    logger.go:42: 09:35:35 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388634 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.388639 | controller |     - '    logger.go:42: 09:35:35 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388644 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:44:27.388648 | controller |     - '    logger.go:42: 09:35:35 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388652 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.388656 | controller |     - '    logger.go:42: 09:35:36 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388661 | controller |       ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.388665 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.388669 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.388674 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_3add:c5d8ab0e913be98c16eb7b7c5c5a1976@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.388678 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.388682 | controller |       driver = messagingv2 transport_url = ''rabbit://default_user_YcxtplLsptl7k25cAVF:F735-UPJOup8HqlYaZyKNnsXxG0GtseU@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.388687 | controller |       ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.388691 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.388695 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:44:27.388705 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:44:27.388709 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.388714 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.388718 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.388722 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.388726 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.388731 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.388735 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.388739 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.388747 | controller |       tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:44:27.388753 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]''
2026-03-14 09:44:27.388760 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.388766 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.388771 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.388777 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.388783 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.388790 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.388796 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.388800 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.388805 | controller |     - '    logger.go:42: 09:35:36 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388809 | controller |       + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.388813 | controller |     - '    logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388818 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.388822 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.388826 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.388831 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.388835 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.388853 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.388858 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.388863 | controller |       == 1 ]'
2026-03-14 09:44:27.388867 | controller |     - '        else'
2026-03-14 09:44:27.388871 | controller |     - '          exit 1'
2026-03-14 09:44:27.388876 | controller |     - '        fi'
2026-03-14 09:44:27.388880 | controller |     - '        ]'
2026-03-14 09:44:27.388884 | controller |     - '    logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388889 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.388893 | controller |     - '    logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388897 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.388902 | controller |     - '    logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388906 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.388910 | controller |     - '    logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388915 | controller |       ++ head -1'
2026-03-14 09:44:27.388919 | controller |     - '    logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388924 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.388930 | controller |     - '    logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388935 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.388939 | controller |     - '    logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388945 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.388950 | controller |     - '    logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388955 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:44:27.388963 | controller |     - '    logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388967 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.388971 | controller |     - '    logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.388983 | controller |       ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.388989 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.388995 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.389001 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_3add:c5d8ab0e913be98c16eb7b7c5c5a1976@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.389010 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.389016 | controller |       driver = messagingv2 transport_url = ''rabbit://default_user_YcxtplLsptl7k25cAVF:F735-UPJOup8HqlYaZyKNnsXxG0GtseU@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.389022 | controller |       ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.389028 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.389034 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:44:27.389040 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:44:27.389046 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.389052 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.389059 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.389065 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.389072 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.389078 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.389084 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.389090 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.389099 | controller |       tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:44:27.389105 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]''
2026-03-14 09:44:27.389111 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.389117 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.389123 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.389129 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.389135 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.389142 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.389147 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.389154 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.389161 | controller |     - '    logger.go:42: 09:35:37 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389165 | controller |       + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.389169 | controller |     - '    logger.go:42: 09:35:38 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389174 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.389178 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.389182 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.389187 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.389191 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.389195 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.389200 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.389204 | controller |       == 1 ]'
2026-03-14 09:44:27.389208 | controller |     - '        else'
2026-03-14 09:44:27.389212 | controller |     - '          exit 1'
2026-03-14 09:44:27.389217 | controller |     - '        fi'
2026-03-14 09:44:27.389221 | controller |     - '        ]'
2026-03-14 09:44:27.389226 | controller |     - '    logger.go:42: 09:35:38 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389231 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.389237 | controller |     - '    logger.go:42: 09:35:38 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389252 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.389259 | controller |     - '    logger.go:42: 09:35:38 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389265 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.389271 | controller |     - '    logger.go:42: 09:35:38 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389276 | controller |       ++ head -1'
2026-03-14 09:44:27.389281 | controller |     - '    logger.go:42: 09:35:38 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389299 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.389308 | controller |     - '    logger.go:42: 09:35:39 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389315 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.389320 | controller |     - '    logger.go:42: 09:35:39 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389324 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.389329 | controller |     - '    logger.go:42: 09:35:39 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389333 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:44:27.389337 | controller |     - '    logger.go:42: 09:35:39 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389342 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.389346 | controller |     - '    logger.go:42: 09:35:39 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389350 | controller |       ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.389355 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.389359 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.389363 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_3add:c5d8ab0e913be98c16eb7b7c5c5a1976@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.389368 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.389374 | controller |       driver = messagingv2 transport_url = ''rabbit://default_user_YcxtplLsptl7k25cAVF:F735-UPJOup8HqlYaZyKNnsXxG0GtseU@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.389378 | controller |       ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.389382 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.389387 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:44:27.389391 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:44:27.389395 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.389400 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.389404 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.389411 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.389416 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.389420 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.389424 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.389429 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.389433 | controller |       tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:44:27.389437 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]''
2026-03-14 09:44:27.389442 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.389446 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.389450 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.389454 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.389459 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.389463 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.389475 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.389480 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.389485 | controller |     - '    logger.go:42: 09:35:39 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389489 | controller |       + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.389494 | controller |     - '    logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389498 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.389502 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.389507 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.389511 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.389515 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.389520 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.389524 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.389540 | controller |       == 1 ]'
2026-03-14 09:44:27.389544 | controller |     - '        else'
2026-03-14 09:44:27.389549 | controller |     - '          exit 1'
2026-03-14 09:44:27.389553 | controller |     - '        fi'
2026-03-14 09:44:27.389557 | controller |     - '        ]'
2026-03-14 09:44:27.389561 | controller |     - '    logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389566 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.389570 | controller |     - '    logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389574 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.389579 | controller |     - '    logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389583 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.389587 | controller |     - '    logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389591 | controller |       ++ head -1'
2026-03-14 09:44:27.389595 | controller |     - '    logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389600 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.389604 | controller |     - '    logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389608 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.389613 | controller |     - '    logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389617 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.389621 | controller |     - '    logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389625 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:44:27.389630 | controller |     - '    logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389634 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.389638 | controller |     - '    logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389642 | controller |       ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.389647 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.389651 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.389655 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_3add:c5d8ab0e913be98c16eb7b7c5c5a1976@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.389660 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.389720 | controller |       driver = messagingv2 transport_url = ''rabbit://default_user_YcxtplLsptl7k25cAVF:F735-UPJOup8HqlYaZyKNnsXxG0GtseU@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.389728 | controller |       ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.389732 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.389736 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:44:27.389741 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:44:27.389745 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.389749 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.389755 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.389760 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.389773 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.389777 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.389782 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.389786 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.389790 | controller |       tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:44:27.389795 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]''
2026-03-14 09:44:27.389799 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.389805 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.389809 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.389813 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.389817 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.389821 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.389826 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.389830 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.389834 | controller |     - '    logger.go:42: 09:35:40 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389838 | controller |       + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.389842 | controller |     - '    logger.go:42: 09:35:41 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389846 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.389851 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.389855 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.389859 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.389863 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.389867 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.389872 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.389876 | controller |       == 1 ]'
2026-03-14 09:44:27.389880 | controller |     - '        else'
2026-03-14 09:44:27.389884 | controller |     - '          exit 1'
2026-03-14 09:44:27.389889 | controller |     - '        fi'
2026-03-14 09:44:27.389893 | controller |     - '        ]'
2026-03-14 09:44:27.389898 | controller |     - '    logger.go:42: 09:35:41 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389902 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.389906 | controller |     - '    logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389911 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.389915 | controller |     - '    logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389920 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.389925 | controller |     - '    logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389932 | controller |       ++ head -1'
2026-03-14 09:44:27.389936 | controller |     - '    logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389942 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.389947 | controller |     - '    logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389951 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.389955 | controller |     - '    logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389960 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.389964 | controller |     - '    logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389968 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:44:27.389972 | controller |     - '    logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389977 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.389981 | controller |     - '    logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.389985 | controller |       ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.389989 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.389993 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.390003 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_3add:c5d8ab0e913be98c16eb7b7c5c5a1976@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.390008 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.390012 | controller |       driver = messagingv2 transport_url = ''rabbit://default_user_YcxtplLsptl7k25cAVF:F735-UPJOup8HqlYaZyKNnsXxG0GtseU@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.390016 | controller |       ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.390020 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.390025 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:44:27.390029 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:44:27.390033 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.390037 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.390041 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.390046 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.390050 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.390054 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.390058 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.390062 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.390067 | controller |       tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:44:27.390071 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]''
2026-03-14 09:44:27.390077 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.390081 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.390085 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.390089 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.390094 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.390098 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.390126 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.390132 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.390137 | controller |     - '    logger.go:42: 09:35:42 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390142 | controller |       + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.390149 | controller |     - '    logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390154 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.390159 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.390164 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.390169 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.390174 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.390179 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.390184 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.390189 | controller |       == 1 ]'
2026-03-14 09:44:27.390194 | controller |     - '        else'
2026-03-14 09:44:27.390199 | controller |     - '          exit 1'
2026-03-14 09:44:27.390205 | controller |     - '        fi'
2026-03-14 09:44:27.390211 | controller |     - '        ]'
2026-03-14 09:44:27.390217 | controller |     - '    logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390223 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.390229 | controller |     - '    logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390235 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.390241 | controller |     - '    logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390246 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.390252 | controller |     - '    logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390258 | controller |       ++ head -1'
2026-03-14 09:44:27.390264 | controller |     - '    logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390268 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.390279 | controller |     - '    logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390283 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.390300 | controller |     - '    logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390307 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.390312 | controller |     - '    logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390317 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:44:27.390321 | controller |     - '    logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390325 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.390332 | controller |     - '    logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390336 | controller |       ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.390518 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.390526 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.390531 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_3add:c5d8ab0e913be98c16eb7b7c5c5a1976@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.390535 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.390539 | controller |       driver = messagingv2 transport_url = ''rabbit://default_user_YcxtplLsptl7k25cAVF:F735-UPJOup8HqlYaZyKNnsXxG0GtseU@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.390544 | controller |       ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.390548 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.390552 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:44:27.390556 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:44:27.390561 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.390565 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.390569 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.390573 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.390578 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.390582 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.390586 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.390590 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.390594 | controller |       tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:44:27.390599 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]''
2026-03-14 09:44:27.390603 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.390607 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.390611 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.390617 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.390622 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.390626 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.390663 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.390669 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.390674 | controller |     - '    logger.go:42: 09:35:43 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390680 | controller |       + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.390685 | controller |     - '    logger.go:42: 09:35:44 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390689 | controller |       running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.390693 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.390708 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.390713 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.390718 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.390722 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.390726 | controller |       |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='')
2026-03-14 09:44:27.390731 | controller |       == 1 ]'
2026-03-14 09:44:27.390735 | controller |     - '        else'
2026-03-14 09:44:27.390739 | controller |     - '          exit 1'
2026-03-14 09:44:27.390743 | controller |     - '        fi'
2026-03-14 09:44:27.390748 | controller |     - '        ]'
2026-03-14 09:44:27.390752 | controller |     - '    logger.go:42: 09:35:44 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390756 | controller |       + oc project watcher-kuttl-default'
2026-03-14 09:44:27.390760 | controller |     - '    logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390765 | controller |       Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.390771 | controller |     - '    logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390775 | controller |       ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.390779 | controller |     - '    logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390783 | controller |       ++ head -1'
2026-03-14 09:44:27.390787 | controller |     - '    logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390792 | controller |       ++ grep -v ''^$'''
2026-03-14 09:44:27.390796 | controller |     - '    logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390800 | controller |       + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.390804 | controller |     - '    logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390808 | controller |       + ''['' -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.390813 | controller |     - '    logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390817 | controller |       ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='''
2026-03-14 09:44:27.390821 | controller |     - '    logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390825 | controller |       +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.390829 | controller |     - '    logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390834 | controller |       ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.390838 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.390842 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.390846 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_3add:c5d8ab0e913be98c16eb7b7c5c5a1976@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.390851 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.390855 | controller |       driver = messagingv2 transport_url = ''rabbit://default_user_YcxtplLsptl7k25cAVF:F735-UPJOup8HqlYaZyKNnsXxG0GtseU@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.390861 | controller |       ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.390865 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.390869 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:44:27.390873 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:44:27.390878 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.390882 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.390886 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.390890 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.390894 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.390898 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.390903 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.390907 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.390911 | controller |       tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:44:27.390921 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]''
2026-03-14 09:44:27.390926 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.390930 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.390934 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.390938 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.390943 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.390947 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.390951 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.390955 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.390959 | controller |     - '    logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390963 | controller |       + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.390968 | controller |     - '    logger.go:42: 09:35:45 | watcher-notification/1-deploy-with-notification |
2026-03-14 09:44:27.390972 | controller |       test step completed 1-deploy-with-notification'
2026-03-14 09:44:27.390976 | controller |     - '    logger.go:42: 09:35:45 | watcher-notification/2-cleanup-watcher | starting
2026-03-14 09:44:27.390980 | controller |       test step 2-cleanup-watcher'
2026-03-14 09:44:27.390984 | controller |     - '    logger.go:42: 09:35:51 | watcher-notification/2-cleanup-watcher | test step
2026-03-14 09:44:27.390989 | controller |       completed 2-cleanup-watcher'
2026-03-14 09:44:27.390993 | controller |     - '    logger.go:42: 09:35:51 | watcher-notification | skipping kubernetes event logging'
2026-03-14 09:44:27.390997 | controller |     - === CONT  kuttl/harness/watcher-topology
2026-03-14 09:44:27.391001 | controller |     - '    logger.go:42: 09:35:51 | watcher-topology | Skipping creation of user-supplied
2026-03-14 09:44:27.391005 | controller |       namespace: watcher-kuttl-default'
2026-03-14 09:44:27.391009 | controller |     - '    logger.go:42: 09:35:51 | watcher-topology/0-cleanup-watcher | starting test
2026-03-14 09:44:27.391015 | controller |       step 0-cleanup-watcher'
2026-03-14 09:44:27.391020 | controller |     - '    logger.go:42: 09:35:51 | watcher-topology/0-cleanup-watcher | test step completed
2026-03-14 09:44:27.391024 | controller |       0-cleanup-watcher'
2026-03-14 09:44:27.391028 | controller |     - '    logger.go:42: 09:35:51 | watcher-topology/1-deploy-with-topology | starting
2026-03-14 09:44:27.391032 | controller |       test step 1-deploy-with-topology'
2026-03-14 09:44:27.391037 | controller |     - '    logger.go:42: 09:35:51 | watcher-topology/1-deploy-with-topology | Topology:watcher-kuttl-default/watcher-api
2026-03-14 09:44:27.391041 | controller |       created'
2026-03-14 09:44:27.391045 | controller |     - '    logger.go:42: 09:35:51 | watcher-topology/1-deploy-with-topology | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:44:27.391049 | controller |       created'
2026-03-14 09:44:27.391054 | controller |     - '    logger.go:42: 09:36:15 | watcher-topology/1-deploy-with-topology | test step
2026-03-14 09:44:27.391058 | controller |       completed 1-deploy-with-topology'
2026-03-14 09:44:27.391062 | controller |     - '    logger.go:42: 09:36:15 | watcher-topology/2-cleanup-watcher | starting test
2026-03-14 09:44:27.391066 | controller |       step 2-cleanup-watcher'
2026-03-14 09:44:27.391072 | controller |     - '    logger.go:42: 09:36:15 | watcher-topology/2-cleanup-watcher | running command:
2026-03-14 09:44:27.391076 | controller |       [sh -c set -ex'
2026-03-14 09:44:27.391080 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.391085 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:44:27.391089 | controller |     - '        ]'
2026-03-14 09:44:27.391093 | controller |     - '    logger.go:42: 09:36:15 | watcher-topology/2-cleanup-watcher | + oc exec -n
2026-03-14 09:44:27.391097 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.391101 | controller |       -c Type'
2026-03-14 09:44:27.391106 | controller |     - '    logger.go:42: 09:36:15 | watcher-topology/2-cleanup-watcher | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.391110 | controller |     - '    logger.go:42: 09:36:18 | watcher-topology/2-cleanup-watcher | + ''['' 0 ==
2026-03-14 09:44:27.391114 | controller |       0 '']'''
2026-03-14 09:44:27.391118 | controller |     - '    logger.go:42: 09:36:19 | watcher-topology/2-cleanup-watcher | running command:
2026-03-14 09:44:27.391123 | controller |       [sh -c set -ex'
2026-03-14 09:44:27.391127 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.391131 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:44:27.391135 | controller |     - '        ]'
2026-03-14 09:44:27.391139 | controller |     - '    logger.go:42: 09:36:19 | watcher-topology/2-cleanup-watcher | + oc exec -n
2026-03-14 09:44:27.391143 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.391148 | controller |       -c Type'
2026-03-14 09:44:27.391152 | controller |     - '    logger.go:42: 09:36:19 | watcher-topology/2-cleanup-watcher | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.391156 | controller |     - '    logger.go:42: 09:36:21 | watcher-topology/2-cleanup-watcher | + ''['' 0 ==
2026-03-14 09:44:27.391160 | controller |       0 '']'''
2026-03-14 09:44:27.391164 | controller |     - '    logger.go:42: 09:36:22 | watcher-topology/2-cleanup-watcher | running command:
2026-03-14 09:44:27.391168 | controller |       [sh -c set -ex'
2026-03-14 09:44:27.391173 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.391182 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:44:27.391187 | controller |     - '        ]'
2026-03-14 09:44:27.391191 | controller |     - '    logger.go:42: 09:36:22 | watcher-topology/2-cleanup-watcher | + oc exec -n
2026-03-14 09:44:27.391195 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.391199 | controller |       -c Type'
2026-03-14 09:44:27.391204 | controller |     - '    logger.go:42: 09:36:22 | watcher-topology/2-cleanup-watcher | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.391208 | controller |     - '    logger.go:42: 09:36:24 | watcher-topology/2-cleanup-watcher | + ''['' 0 ==
2026-03-14 09:44:27.391214 | controller |       0 '']'''
2026-03-14 09:44:27.391218 | controller |     - '    logger.go:42: 09:36:24 | watcher-topology/2-cleanup-watcher | test step completed
2026-03-14 09:44:27.391222 | controller |       2-cleanup-watcher'
2026-03-14 09:44:27.391227 | controller |     - '    logger.go:42: 09:36:24 | watcher-topology | skipping kubernetes event logging'
2026-03-14 09:44:27.391231 | controller |     - === CONT  kuttl/harness/watcher-tls-certs-change
2026-03-14 09:44:27.391235 | controller |     - '    logger.go:42: 09:36:24 | watcher-tls-certs-change | Skipping creation of user-supplied
2026-03-14 09:44:27.391239 | controller |       namespace: watcher-kuttl-default'
2026-03-14 09:44:27.391243 | controller |     - '    logger.go:42: 09:36:24 | watcher-tls-certs-change/0-cleanup-watcher | starting
2026-03-14 09:44:27.391248 | controller |       test step 0-cleanup-watcher'
2026-03-14 09:44:27.391252 | controller |     - '    logger.go:42: 09:36:24 | watcher-tls-certs-change/0-cleanup-watcher | test
2026-03-14 09:44:27.391256 | controller |       step completed 0-cleanup-watcher'
2026-03-14 09:44:27.391260 | controller |     - '    logger.go:42: 09:36:24 | watcher-tls-certs-change/1-deploy-with-tlse | starting
2026-03-14 09:44:27.391264 | controller |       test step 1-deploy-with-tlse'
2026-03-14 09:44:27.391269 | controller |     - '    logger.go:42: 09:36:24 | watcher-tls-certs-change/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-internal-svc
2026-03-14 09:44:27.391273 | controller |       created'
2026-03-14 09:44:27.391277 | controller |     - '    logger.go:42: 09:36:24 | watcher-tls-certs-change/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc
2026-03-14 09:44:27.391281 | controller |       created'
2026-03-14 09:44:27.391288 | controller |     - '    logger.go:42: 09:36:24 | watcher-tls-certs-change/1-deploy-with-tlse | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:44:27.391308 | controller |       created'
2026-03-14 09:44:27.391312 | controller |     - '    logger.go:42: 09:36:25 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:44:27.391317 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.391321 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.391325 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:44:27.391329 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.391334 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.391338 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.391342 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:44:27.391346 | controller |     - '          exit 1'
2026-03-14 09:44:27.391350 | controller |     - '        fi'
2026-03-14 09:44:27.391355 | controller |     - '        '
2026-03-14 09:44:27.391359 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.391363 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.391367 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.391372 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.391378 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.391383 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:44:27.391389 | controller |     - '          exit 1'
2026-03-14 09:44:27.391395 | controller |     - '        fi'
2026-03-14 09:44:27.391401 | controller |     - '        ]'
2026-03-14 09:44:27.391407 | controller |     - '    logger.go:42: 09:36:25 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.391413 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:44:27.391422 | controller |     - '    logger.go:42: 09:36:25 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.391427 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:44:27.391433 | controller |     - '    logger.go:42: 09:36:25 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:44:27.391441 | controller |     - '    logger.go:42: 09:36:26 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:44:27.391446 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.391452 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.391458 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:44:27.391464 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.391470 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.391475 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.391479 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:44:27.391483 | controller |     - '          exit 1'
2026-03-14 09:44:27.391488 | controller |     - '        fi'
2026-03-14 09:44:27.391492 | controller |     - '        '
2026-03-14 09:44:27.391496 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.391500 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.391504 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.391517 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.391521 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.391526 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:44:27.391530 | controller |     - '          exit 1'
2026-03-14 09:44:27.391534 | controller |     - '        fi'
2026-03-14 09:44:27.391538 | controller |     - '        ]'
2026-03-14 09:44:27.391543 | controller |     - '    logger.go:42: 09:36:26 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.391547 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:44:27.391553 | controller |     - '    logger.go:42: 09:36:26 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.391558 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:44:27.391562 | controller |     - '    logger.go:42: 09:36:26 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:44:27.391566 | controller |     - '    logger.go:42: 09:36:27 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:44:27.391570 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.391574 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.391579 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:44:27.391583 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.391587 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.391591 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.391595 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:44:27.391600 | controller |     - '          exit 1'
2026-03-14 09:44:27.391604 | controller |     - '        fi'
2026-03-14 09:44:27.391608 | controller |     - '        '
2026-03-14 09:44:27.391612 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.391616 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.391620 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.391625 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.391629 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.391633 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:44:27.391637 | controller |     - '          exit 1'
2026-03-14 09:44:27.391641 | controller |     - '        fi'
2026-03-14 09:44:27.391646 | controller |     - '        ]'
2026-03-14 09:44:27.391650 | controller |     - '    logger.go:42: 09:36:27 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.391656 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:44:27.391660 | controller |     - '    logger.go:42: 09:36:27 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.391664 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:44:27.391669 | controller |     - '    logger.go:42: 09:36:27 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:44:27.391673 | controller |     - '    logger.go:42: 09:36:28 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:44:27.391677 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.391681 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.391685 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:44:27.391690 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.391694 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.391698 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.391702 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:44:27.391707 | controller |     - '          exit 1'
2026-03-14 09:44:27.391711 | controller |     - '        fi'
2026-03-14 09:44:27.391715 | controller |     - '        '
2026-03-14 09:44:27.391719 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.391723 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.391727 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.391732 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.391736 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.391740 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:44:27.391744 | controller |     - '          exit 1'
2026-03-14 09:44:27.391748 | controller |     - '        fi'
2026-03-14 09:44:27.391753 | controller |     - '        ]'
2026-03-14 09:44:27.391757 | controller |     - '    logger.go:42: 09:36:28 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.391761 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:44:27.391767 | controller |     - '    logger.go:42: 09:36:28 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.391771 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:44:27.391775 | controller |     - '    logger.go:42: 09:36:28 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:44:27.391780 | controller |     - '    logger.go:42: 09:36:29 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:44:27.391812 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.391818 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.391822 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:44:27.391833 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.391837 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.391841 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.391846 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:44:27.391850 | controller |     - '          exit 1'
2026-03-14 09:44:27.391854 | controller |     - '        fi'
2026-03-14 09:44:27.391858 | controller |     - '        '
2026-03-14 09:44:27.391863 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.391867 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.391871 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.391875 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.391882 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.391886 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:44:27.391890 | controller |     - '          exit 1'
2026-03-14 09:44:27.391895 | controller |     - '        fi'
2026-03-14 09:44:27.391899 | controller |     - '        ]'
2026-03-14 09:44:27.391903 | controller |     - '    logger.go:42: 09:36:29 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.391907 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:44:27.391915 | controller |     - '    logger.go:42: 09:36:30 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.391919 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:44:27.391923 | controller |     - '    logger.go:42: 09:36:30 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:44:27.391929 | controller |     - '    logger.go:42: 09:36:31 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:44:27.391933 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.391937 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.391941 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:44:27.391946 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.391952 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.391956 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.391960 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:44:27.391965 | controller |     - '          exit 1'
2026-03-14 09:44:27.391969 | controller |     - '        fi'
2026-03-14 09:44:27.391973 | controller |     - '        '
2026-03-14 09:44:27.391977 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.391982 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.391986 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.391990 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.391994 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.391999 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:44:27.392003 | controller |     - '          exit 1'
2026-03-14 09:44:27.392007 | controller |     - '        fi'
2026-03-14 09:44:27.392011 | controller |     - '        ]'
2026-03-14 09:44:27.392016 | controller |     - '    logger.go:42: 09:36:31 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.392020 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:44:27.392025 | controller |     - '    logger.go:42: 09:36:31 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.392030 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:44:27.392034 | controller |     - '    logger.go:42: 09:36:31 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:44:27.392038 | controller |     - '    logger.go:42: 09:36:32 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:44:27.392042 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.392047 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.392051 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:44:27.392055 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.392060 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.392064 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.392068 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:44:27.392072 | controller |     - '          exit 1'
2026-03-14 09:44:27.392079 | controller |     - '        fi'
2026-03-14 09:44:27.392084 | controller |     - '        '
2026-03-14 09:44:27.392088 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.392092 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.392096 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.392101 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.392105 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.392109 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:44:27.392113 | controller |     - '          exit 1'
2026-03-14 09:44:27.392117 | controller |     - '        fi'
2026-03-14 09:44:27.392122 | controller |     - '        ]'
2026-03-14 09:44:27.392126 | controller |     - '    logger.go:42: 09:36:32 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.392136 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:44:27.392143 | controller |     - '    logger.go:42: 09:36:32 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.392147 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:44:27.392151 | controller |     - '    logger.go:42: 09:36:32 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:44:27.392156 | controller |     - '    logger.go:42: 09:36:33 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:44:27.392160 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.392164 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.392168 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:44:27.392172 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.392176 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.392181 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.392185 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:44:27.392189 | controller |     - '          exit 1'
2026-03-14 09:44:27.392193 | controller |     - '        fi'
2026-03-14 09:44:27.392197 | controller |     - '        '
2026-03-14 09:44:27.392202 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.392206 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.392210 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.392214 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.392218 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.392222 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:44:27.392227 | controller |     - '          exit 1'
2026-03-14 09:44:27.392231 | controller |     - '        fi'
2026-03-14 09:44:27.392235 | controller |     - '        ]'
2026-03-14 09:44:27.392239 | controller |     - '    logger.go:42: 09:36:33 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.392243 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:44:27.392248 | controller |     - '    logger.go:42: 09:36:33 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.392252 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:44:27.392256 | controller |     - '    logger.go:42: 09:36:33 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:44:27.392260 | controller |     - '    logger.go:42: 09:36:34 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:44:27.392264 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.392269 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.392273 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:44:27.392279 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.392283 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.392301 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.392308 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:44:27.392313 | controller |     - '          exit 1'
2026-03-14 09:44:27.392317 | controller |     - '        fi'
2026-03-14 09:44:27.392321 | controller |     - '        '
2026-03-14 09:44:27.392325 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.392329 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.392334 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.392338 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.392342 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.392346 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:44:27.392350 | controller |     - '          exit 1'
2026-03-14 09:44:27.392377 | controller |     - '        fi'
2026-03-14 09:44:27.392384 | controller |     - '        ]'
2026-03-14 09:44:27.392389 | controller |     - '    logger.go:42: 09:36:34 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.392393 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:44:27.392397 | controller |     - '    logger.go:42: 09:36:34 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.392402 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:44:27.392406 | controller |     - '    logger.go:42: 09:36:34 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:44:27.392410 | controller |     - '    logger.go:42: 09:36:36 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:44:27.392414 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.392418 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.392423 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:44:27.392427 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.392431 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.392435 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.392439 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:44:27.392443 | controller |     - '          exit 1'
2026-03-14 09:44:27.392448 | controller |     - '        fi'
2026-03-14 09:44:27.392452 | controller |     - '        '
2026-03-14 09:44:27.392463 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.392467 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.392472 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.392476 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.392480 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.392484 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:44:27.392488 | controller |     - '          exit 1'
2026-03-14 09:44:27.392493 | controller |     - '        fi'
2026-03-14 09:44:27.392497 | controller |     - '        ]'
2026-03-14 09:44:27.392501 | controller |     - '    logger.go:42: 09:36:36 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.392505 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:44:27.392512 | controller |     - '    logger.go:42: 09:36:36 | watcher-tls-certs-change/1-deploy-with-tlse | Error
2026-03-14 09:44:27.392516 | controller |       from server (NotFound): pods "watcher-kuttl-api-0" not found'
2026-03-14 09:44:27.392521 | controller |     - '    logger.go:42: 09:36:36 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='
2026-03-14 09:44:27.392527 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:44:27.392531 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.392535 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.392540 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:44:27.392544 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.392548 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.392552 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.392556 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:44:27.392560 | controller |     - '          exit 1'
2026-03-14 09:44:27.392565 | controller |     - '        fi'
2026-03-14 09:44:27.392569 | controller |     - '        '
2026-03-14 09:44:27.392573 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.392577 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.392581 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.392586 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.392590 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.392594 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:44:27.392598 | controller |     - '          exit 1'
2026-03-14 09:44:27.392602 | controller |     - '        fi'
2026-03-14 09:44:27.392607 | controller |     - '        ]'
2026-03-14 09:44:27.392611 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.392615 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:44:27.392619 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN
2026-03-14 09:44:27.392623 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.392628 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.392632 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.392636 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.392641 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.392645 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.392649 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.392653 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.392657 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.392662 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.392666 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.392670 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.392676 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.392686 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.392691 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.392695 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.392700 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.392704 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.392708 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.392712 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.392719 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.392723 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.392727 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.392732 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:44:27.392736 | controller |       --decode'
2026-03-14 09:44:27.392740 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.392744 | controller |       get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:44:27.392749 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN
2026-03-14 09:44:27.392753 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.392757 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.392761 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.392765 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.392769 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.392774 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.392778 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.392782 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.392786 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.392790 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.392795 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.392801 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.392805 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.392809 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.392813 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.392817 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.392822 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.392833 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.392838 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.392843 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.392848 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.392852 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.392856 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.392860 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:44:27.392865 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.392890 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.392895 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.392900 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.392904 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.392909 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.392913 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.392917 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.392921 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.392925 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.392929 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.392934 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.392940 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.392944 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.392949 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.392953 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.392957 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.392962 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.392966 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.392974 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.392979 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.392983 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.392988 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.392993 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.393004 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.393009 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.393013 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.393018 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.393022 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.393026 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.393030 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.393034 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.393039 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.393043 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.393047 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.393051 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.393057 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.393062 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.393066 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.393070 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.393074 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.393078 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.393083 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.393087 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.393091 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:44:27.393095 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.393100 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:44:27.393104 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN
2026-03-14 09:44:27.393108 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.393112 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.393119 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.393123 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.393128 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.393132 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.393141 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.393146 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.393150 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.393154 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.393159 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.393163 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.393167 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.393173 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.393177 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.393181 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.393186 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.393190 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.393194 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.393198 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.393202 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.393207 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.393211 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.393215 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:44:27.393219 | controller |       --decode'
2026-03-14 09:44:27.393224 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.393228 | controller |       get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:44:27.393232 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN
2026-03-14 09:44:27.393236 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.393240 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.393245 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.393252 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.393256 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.393261 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.393265 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.393269 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.393273 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.393277 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.393288 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.393305 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.393312 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.393316 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.393321 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.393325 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.393329 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.393333 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.393337 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.393342 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.393346 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.393350 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.393354 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.393359 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:44:27.393363 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.393367 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.393371 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.393375 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.393386 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.393391 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.393395 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.393399 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.393403 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.393409 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.393413 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.393417 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.393421 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.393441 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.393446 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.393450 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.393461 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.393466 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.393471 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.393475 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.393479 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.393483 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.393488 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.393492 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.393496 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.393500 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.393505 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.393511 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.393516 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.393520 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.393524 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.393528 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.393533 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.393537 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.393541 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.393545 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.393550 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.393556 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.393560 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.393564 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.393568 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.393573 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.393577 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.393581 | controller |     - '    logger.go:42: 09:36:37 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.393585 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:44:27.393590 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:44:27.393594 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.393598 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.393609 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:44:27.393614 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.393619 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.393623 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.393627 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:44:27.393631 | controller |     - '          exit 1'
2026-03-14 09:44:27.393636 | controller |     - '        fi'
2026-03-14 09:44:27.393640 | controller |     - '        '
2026-03-14 09:44:27.393645 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.393649 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.393653 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.393657 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.393662 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.393666 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:44:27.393670 | controller |     - '          exit 1'
2026-03-14 09:44:27.393675 | controller |     - '        fi'
2026-03-14 09:44:27.393679 | controller |     - '        ]'
2026-03-14 09:44:27.393683 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.393687 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:44:27.393692 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN
2026-03-14 09:44:27.393696 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.393700 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.393705 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.393709 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.393713 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.393719 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.393725 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.393729 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.393734 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.393738 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.393742 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.393746 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.393750 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.393755 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.393759 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.393763 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.393767 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.393771 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.393776 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.393780 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.393789 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.393794 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.393798 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.393802 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:44:27.393807 | controller |       --decode'
2026-03-14 09:44:27.393811 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.393815 | controller |       get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:44:27.393819 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN
2026-03-14 09:44:27.393823 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.393828 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.393832 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.393836 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.393843 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.393847 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.393851 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.393856 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.393861 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.393866 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.393870 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.393874 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.393878 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.393882 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.393887 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.393891 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.393895 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.393899 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.393903 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.393907 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.393912 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.393916 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.393920 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.393926 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:44:27.393930 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.393934 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.393939 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.393948 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.393952 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.393959 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.393965 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.393969 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.393974 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.393979 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.393984 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.393990 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.393995 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.393999 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.394003 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.394007 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.394012 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.394016 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.394021 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.394027 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.394033 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.394039 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.394045 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.394051 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.394057 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.394063 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.394069 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.394075 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.394081 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.394088 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.394093 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.394103 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.394108 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.394127 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.394131 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.394139 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.394144 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.394148 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.394152 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.394157 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.394161 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.394165 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.394169 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.394174 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.394178 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:44:27.394182 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.394186 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:44:27.394191 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN
2026-03-14 09:44:27.394195 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.394199 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.394203 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.394208 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.394212 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.394216 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.394220 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.394226 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.394231 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.394235 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.394239 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.394243 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.394247 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.394252 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.394263 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.394267 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.394272 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.394276 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.394280 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.394284 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.394301 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.394308 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.394313 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.394317 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:44:27.394321 | controller |       --decode'
2026-03-14 09:44:27.394325 | controller |     - '    logger.go:42: 09:36:39 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.394330 | controller |       get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:44:27.394334 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN
2026-03-14 09:44:27.394338 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.394342 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.394347 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.394351 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.394355 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.394359 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.394366 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.394370 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.394374 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.394378 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.394383 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.394387 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.394391 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.394395 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.394400 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.394406 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.394410 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.394414 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.394424 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.394429 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.394433 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.394437 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.394441 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.394446 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:44:27.394450 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.394454 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.394458 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.394463 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.394467 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.394471 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.394475 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.394481 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.394485 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.394490 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.394494 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.394498 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.394502 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.394506 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.394511 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.394515 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.394521 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.394525 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.394529 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.394533 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.394538 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.394542 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.394546 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.394550 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.394554 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.394563 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.394568 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.394572 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.394576 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.394580 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.394585 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.394591 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.394595 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.394599 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.394603 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.394607 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.394612 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.394616 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.394620 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.394626 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.394631 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.394637 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.394642 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.394648 | controller |     - '    logger.go:42: 09:36:40 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.394653 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:44:27.394659 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:44:27.394665 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.394670 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.394676 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:44:27.394682 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.394688 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.394693 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.394699 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:44:27.394704 | controller |     - '          exit 1'
2026-03-14 09:44:27.394710 | controller |     - '        fi'
2026-03-14 09:44:27.394715 | controller |     - '        '
2026-03-14 09:44:27.394721 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.394726 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.394731 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.394737 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.394742 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.394748 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:44:27.394754 | controller |     - '          exit 1'
2026-03-14 09:44:27.394760 | controller |     - '        fi'
2026-03-14 09:44:27.394765 | controller |     - '        ]'
2026-03-14 09:44:27.394773 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.394779 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:44:27.394791 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN
2026-03-14 09:44:27.394798 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.394803 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.394809 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.394814 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.394820 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.394825 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.394831 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.394836 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.394842 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.394848 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.394853 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.394859 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.394865 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.394870 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.394876 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.394882 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.394887 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.394893 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.394902 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.394908 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.394913 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.394919 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.394927 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.394932 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:44:27.394938 | controller |       --decode'
2026-03-14 09:44:27.394944 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.394949 | controller |       get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:44:27.394955 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN
2026-03-14 09:44:27.394960 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.394966 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.394972 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.394978 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.394993 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.394999 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.395005 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.395011 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.395016 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.395022 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.395027 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.395033 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.395038 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.395044 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.395050 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.395055 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.395061 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.395067 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.395073 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.395078 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.395084 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.395092 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.395098 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.395104 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:44:27.395113 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.395119 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.395125 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.395130 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.395136 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.395141 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.395147 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.395153 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.395158 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.395164 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.395176 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.395183 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.395187 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.395191 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.395195 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.395200 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.395204 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.395208 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.395212 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.395217 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.395221 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.395225 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.395233 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.395237 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.395241 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.395248 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.395252 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.395256 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.395260 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.395264 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.395269 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.395273 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.395277 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.395281 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.395288 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.395307 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.395312 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.395316 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.395320 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.395330 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.395334 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.395339 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.395343 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.395347 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.395351 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:44:27.395356 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.395360 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:44:27.395364 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN
2026-03-14 09:44:27.395370 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.395375 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.395379 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.395383 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.395387 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.395395 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.395399 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.395403 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.395407 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.395411 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.395416 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.395420 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.395424 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.395428 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.395432 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.395436 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.395441 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.395445 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.395449 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.395455 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.395459 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.395463 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.395467 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.395477 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:44:27.395482 | controller |       --decode'
2026-03-14 09:44:27.395486 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.395492 | controller |       get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:44:27.395496 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN
2026-03-14 09:44:27.395500 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.395505 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.395509 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.395513 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.395517 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.395521 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.395527 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.395532 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.395536 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.395540 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.395544 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.395548 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.395553 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.395557 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.395561 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.395565 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.395569 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.395573 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.395578 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.395582 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.395586 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.395590 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.395594 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.395601 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:44:27.395605 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.395609 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.395613 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.395618 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.395627 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.395632 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.395636 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.395642 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.395646 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.395650 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.395655 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.395659 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.395663 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.395667 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.395671 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.395675 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.395680 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.395684 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.395688 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.395692 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.395696 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.395701 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.395705 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.395709 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.395724 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.395728 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.395733 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.395737 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.395741 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.395745 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.395752 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.395756 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.395760 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.395769 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.395774 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.395779 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.395783 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.395787 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.395791 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.395795 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.395800 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.395804 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.395808 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.395812 | controller |     - '    logger.go:42: 09:36:41 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.395817 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:44:27.395821 | controller |     - '    logger.go:42: 09:36:42 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:44:27.395825 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.395829 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.395833 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:44:27.395838 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.395842 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.395848 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.395852 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:44:27.395857 | controller |     - '          exit 1'
2026-03-14 09:44:27.395861 | controller |     - '        fi'
2026-03-14 09:44:27.395865 | controller |     - '        '
2026-03-14 09:44:27.395869 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.395873 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.395878 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.395882 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.395886 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.395890 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:44:27.395894 | controller |     - '          exit 1'
2026-03-14 09:44:27.395899 | controller |     - '        fi'
2026-03-14 09:44:27.395903 | controller |     - '        ]'
2026-03-14 09:44:27.395907 | controller |     - '    logger.go:42: 09:36:42 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.395912 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:44:27.395916 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN
2026-03-14 09:44:27.395920 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.395924 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.395928 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.395933 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.395937 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.395941 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.395945 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.395949 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.395958 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.395966 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.395970 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.395974 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.395978 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.395983 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.395987 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.395993 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.395997 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.396001 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.396006 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.396010 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.396014 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.396018 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.396023 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.396027 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:44:27.396031 | controller |       --decode'
2026-03-14 09:44:27.396035 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.396039 | controller |       get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:44:27.396044 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN
2026-03-14 09:44:27.396048 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.396052 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.396056 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.396060 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.396065 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.396069 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.396073 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.396077 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.396082 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.396086 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.396090 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.396096 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.396105 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.396109 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.396116 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.396121 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.396125 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.396129 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.396133 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.396138 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.396142 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.396146 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.396150 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.396155 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:44:27.396159 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.396163 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.396167 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.396171 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.396176 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.396180 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.396184 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.396188 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.396192 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.396197 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.396201 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.396205 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.396209 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.396215 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.396219 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.396234 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.396239 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.396243 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.396252 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.396256 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.396260 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.396265 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.396269 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.396273 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.396277 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.396281 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.396289 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.396305 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.396310 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.396314 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.396319 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.396323 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.396327 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.396331 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.396336 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.396340 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.396344 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.396351 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.396355 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.396361 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.396365 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.396369 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.396374 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.396378 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.396382 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:44:27.396518 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.396523 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:44:27.396527 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN
2026-03-14 09:44:27.396532 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.396536 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.396548 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.396553 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.396557 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.396561 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.396566 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.396570 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.396574 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.396578 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.396583 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.396587 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.396591 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.396595 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.396599 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.396603 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.396608 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.396616 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.396621 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.396625 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.396630 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.396635 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.396641 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.396647 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:44:27.396653 | controller |       --decode'
2026-03-14 09:44:27.396659 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.396664 | controller |       get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:44:27.396668 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN
2026-03-14 09:44:27.396673 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.396677 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.396681 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.396685 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.396689 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.396694 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.396704 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.396709 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.396713 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.396717 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.396721 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.396725 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.396730 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.396734 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.396738 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.396742 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.396749 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.396753 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.396759 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.396764 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.396768 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.396772 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.396776 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.396780 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:44:27.396785 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.396789 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.396793 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.396797 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.396802 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.396806 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.396810 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.396814 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.396818 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.396822 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.396827 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.396831 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.396842 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.396847 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.396851 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.396855 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.396860 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.396865 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.396870 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.396874 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.396878 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.396885 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.396889 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.396893 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.396898 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.396902 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.396906 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.396911 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.396915 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.396920 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.396925 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.396930 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.396934 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.396939 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.396944 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.396949 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.396953 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.396958 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.396962 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.396966 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.396975 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.396982 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.396988 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.396993 | controller |     - '    logger.go:42: 09:36:43 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.396997 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:44:27.397003 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:44:27.397007 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.397012 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.397018 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:44:27.397024 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.397029 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.397035 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.397044 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:44:27.397049 | controller |     - '          exit 1'
2026-03-14 09:44:27.397053 | controller |     - '        fi'
2026-03-14 09:44:27.397058 | controller |     - '        '
2026-03-14 09:44:27.397063 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.397068 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.397075 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.397080 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.397086 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.397092 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:44:27.397098 | controller |     - '          exit 1'
2026-03-14 09:44:27.397104 | controller |     - '        fi'
2026-03-14 09:44:27.397110 | controller |     - '        ]'
2026-03-14 09:44:27.397115 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.397121 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:44:27.397127 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN
2026-03-14 09:44:27.397133 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.397140 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.397146 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.397152 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.397158 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.397165 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.397171 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.397177 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.397183 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.397192 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.397197 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.397201 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.397205 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.397210 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.397214 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.397226 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.397231 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.397235 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.397239 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.397243 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.397248 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.397252 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.397260 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.397264 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.397269 | controller |       get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:44:27.397273 | controller |     - '    logger.go:42: 09:36:44 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:44:27.397279 | controller |       --decode'
2026-03-14 09:44:27.397283 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN
2026-03-14 09:44:27.397305 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.397313 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.397319 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.397324 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.397328 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.397332 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.397336 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.397340 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.397346 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.397351 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.397356 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.397360 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.397364 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.397368 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.397372 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.397377 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.397381 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.397385 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.397389 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.397399 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.397404 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.397409 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.397413 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.397417 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:44:27.397421 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.397426 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.397430 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.397434 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.397440 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.397444 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.397448 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.397453 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.397457 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.397464 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.397468 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.397472 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.397476 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.397481 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.397485 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.397489 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.397493 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.397497 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.397502 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.397506 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.397510 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.397514 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.397518 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.397523 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.397527 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.397531 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.397535 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.397545 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.397552 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.397556 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.397560 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.397565 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.397569 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.397575 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.397579 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.397583 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.397587 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.397591 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.397596 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.397600 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.397604 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.397608 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.397612 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.397617 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.397621 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:44:27.397625 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.397629 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:44:27.397634 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN
2026-03-14 09:44:27.397638 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.397642 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.397646 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.397650 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.397655 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.397659 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.397663 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.397667 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.397673 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.397683 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.397687 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.397694 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.397699 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.397703 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.397707 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.397711 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.397715 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.397719 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.397724 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.397728 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.397732 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.397736 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.397740 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.397745 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:44:27.397749 | controller |       --decode'
2026-03-14 09:44:27.397753 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.397758 | controller |       get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:44:27.397762 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN
2026-03-14 09:44:27.397767 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.397771 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.397775 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.397780 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.397784 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.397788 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.397792 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.397796 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.397801 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.397808 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.397812 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.397817 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.397821 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.397830 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.397835 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.397839 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.397843 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.397848 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.397852 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.397856 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.397860 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.397864 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.397869 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.397873 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:44:27.397877 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.397881 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.397886 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.397890 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.397894 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.397898 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.397902 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.397907 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.397911 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.397915 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.397924 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.397928 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.397933 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.397937 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.397941 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.397945 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.397949 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.397954 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.397958 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.397967 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.397971 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.397976 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.397980 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.397984 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.397988 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.397993 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.397997 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.398001 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.398005 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.398009 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.398014 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.398018 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.398022 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.398028 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.398034 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.398038 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.398043 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.398047 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.398051 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.398057 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.398061 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.398065 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.398069 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.398074 | controller |     - '    logger.go:42: 09:36:45 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.398078 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:44:27.398082 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:44:27.398086 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.398091 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.398095 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:44:27.398101 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.398105 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.398109 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.398114 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:44:27.398135 | controller |     - '          exit 1'
2026-03-14 09:44:27.398142 | controller |     - '        fi'
2026-03-14 09:44:27.398147 | controller |     - '        '
2026-03-14 09:44:27.398159 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.398165 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.398171 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.398177 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.398183 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.398188 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:44:27.398194 | controller |     - '          exit 1'
2026-03-14 09:44:27.398199 | controller |     - '        fi'
2026-03-14 09:44:27.398205 | controller |     - '        ]'
2026-03-14 09:44:27.398211 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.398217 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:44:27.398223 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN
2026-03-14 09:44:27.398229 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.398236 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.398245 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.398251 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.398257 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.398262 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.398268 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.398274 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.398280 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.398299 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.398307 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.398312 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.398319 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.398323 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.398328 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.398332 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.398336 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.398340 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.398344 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.398349 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.398353 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.398359 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.398363 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.398368 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.398372 | controller |       get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:44:27.398386 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:44:27.398391 | controller |       --decode'
2026-03-14 09:44:27.398395 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN
2026-03-14 09:44:27.398402 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.398408 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.398412 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.398416 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.398421 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.398425 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.398429 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.398433 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.398438 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.398442 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.398446 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.398450 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.398455 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.398459 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.398463 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.398469 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.398473 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.398478 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.398482 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.398486 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.398490 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.398495 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.398499 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.398503 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:44:27.398508 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.398512 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.398518 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.398522 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.398527 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.398536 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.398541 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.398547 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.398553 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.398559 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.398564 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.398570 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.398576 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.398581 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.398587 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.398592 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.398599 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.398603 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.398608 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.398612 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.398616 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.398620 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.398625 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.398629 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.398634 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.398639 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.398648 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.398653 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.398659 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.398665 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.398670 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.398676 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.398681 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.398688 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.398702 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.398709 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.398715 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.398719 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.398723 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.398728 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.398736 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.398742 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.398752 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.398758 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.398763 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:44:27.398769 | controller |     - '    logger.go:42: 09:36:46 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.398774 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:44:27.398780 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN
2026-03-14 09:44:27.398786 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.398792 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.398797 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.398802 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.398809 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.398813 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.398817 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.398821 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.398825 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.398830 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.398834 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.398838 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.398842 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.398846 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.398850 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.398855 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.398859 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.398870 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.398875 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.398879 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.398886 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.398891 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.398895 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.398899 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.398903 | controller |       get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:44:27.398908 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:44:27.398912 | controller |       --decode'
2026-03-14 09:44:27.398916 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN
2026-03-14 09:44:27.398920 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.398924 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.398929 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.398935 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.398939 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.398943 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.398947 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.398952 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.398956 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.398960 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.398964 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.398968 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.398972 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.398977 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.398981 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.398985 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.398989 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.398993 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.398998 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.399002 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.399006 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.399017 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.399022 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.399026 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:44:27.399031 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.399035 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.399039 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.399043 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.399049 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.399053 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.399057 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.399061 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.399066 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.399070 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.399074 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.399078 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.399082 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.399087 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.399091 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.399095 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.399099 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.399103 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.399107 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.399112 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.399116 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.399120 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.399124 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.399128 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.399134 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.399139 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.399143 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.399147 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.399158 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.399163 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.399167 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.399171 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.399175 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.399180 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.399184 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.399188 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.399192 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.399196 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.399200 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.399205 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.399209 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.399213 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.399217 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.399221 | controller |     - '    logger.go:42: 09:36:47 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.399226 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:44:27.399230 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | running
2026-03-14 09:44:27.399234 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.399238 | controller |     - '        public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.399242 | controller |       cat /etc/pki/tls/certs/public.crt)'
2026-03-14 09:44:27.399247 | controller |     - '        public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc
2026-03-14 09:44:27.399251 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.399255 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.399259 | controller |     - '        if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then'
2026-03-14 09:44:27.399264 | controller |     - '          exit 1'
2026-03-14 09:44:27.399268 | controller |     - '        fi'
2026-03-14 09:44:27.399272 | controller |     - '        '
2026-03-14 09:44:27.399276 | controller |     - '        internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0
2026-03-14 09:44:27.399280 | controller |       cat /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.399287 | controller |     - '        internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc
2026-03-14 09:44:27.399308 | controller |       -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.399313 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.399317 | controller |     - '        if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then'
2026-03-14 09:44:27.399321 | controller |     - '          exit 1'
2026-03-14 09:44:27.399325 | controller |     - '        fi'
2026-03-14 09:44:27.399329 | controller |     - '        ]'
2026-03-14 09:44:27.399334 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.399338 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:44:27.399342 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN
2026-03-14 09:44:27.399346 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.399350 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.399355 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.399367 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.399371 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.399376 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.399380 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.399384 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.399388 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.399392 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.399396 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.399401 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.399405 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.399409 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.399413 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.399417 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.399422 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.399426 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.399430 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.399436 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.399440 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.399444 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.399449 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.399453 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:44:27.399457 | controller |       --decode'
2026-03-14 09:44:27.399461 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.399466 | controller |       get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:44:27.399470 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN
2026-03-14 09:44:27.399474 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.399478 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.399482 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.399487 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.399491 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.399497 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.399501 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.399510 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.399514 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.399518 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.399523 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.399527 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.399531 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.399535 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.399539 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.399543 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.399548 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.399552 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.399558 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.399562 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.399566 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.399570 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.399575 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.399579 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:44:27.399584 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.399589 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.399600 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.399613 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.399620 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.399626 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.399632 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.399642 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.399648 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.399654 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.399660 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.399665 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.399671 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.399686 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.399693 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.399698 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.399702 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.399706 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.399710 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.399717 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.399721 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.399726 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.399730 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.399734 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.399738 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.399742 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.399747 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.399751 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.399755 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.399759 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.399763 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.399772 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.399776 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.399780 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.399784 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.399788 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.399792 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.399797 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.399801 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.399805 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.399809 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.399813 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.399823 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.399829 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.399833 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:44:27.399838 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.399842 | controller |       rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:44:27.399846 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN
2026-03-14 09:44:27.399850 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.399855 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.399859 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.399863 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.399867 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.399871 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.399875 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.399880 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.399884 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.399890 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.399895 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.399904 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.399910 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.399915 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.399921 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.399927 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.399932 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.399938 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.399943 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.399949 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.399954 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.399962 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.399968 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.399977 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64
2026-03-14 09:44:27.399983 | controller |       --decode'
2026-03-14 09:44:27.399989 | controller |     - '    logger.go:42: 09:36:48 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc
2026-03-14 09:44:27.399994 | controller |       get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:44:27.400000 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN
2026-03-14 09:44:27.400006 | controller |       CERTIFICATE-----'
2026-03-14 09:44:27.400018 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.400025 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.400031 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.400037 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.400043 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.400049 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.400054 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.400060 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.400066 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.400072 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.400077 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.400087 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.400092 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.400098 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.400104 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.400109 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.400115 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.400121 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.400129 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.400135 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.400141 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.400147 | controller |       CERTIFICATE-----'''
2026-03-14 09:44:27.400153 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | + ''[''
2026-03-14 09:44:27.400158 | controller |       ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.400164 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.400169 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.400175 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.400180 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.400186 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.400191 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.400204 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.400211 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.400216 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.400222 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.400228 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.400233 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.400242 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.400248 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.400254 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.400260 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.400265 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.400271 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.400277 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.400303 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.400312 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.400316 | controller |       CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.400321 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRAOviQaqjjQ4A8GTn0ObYN9UwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.400325 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMxNDA5MzYyNloXDTMx'
2026-03-14 09:44:27.400329 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxMzA5MzYyNlowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFm'
2026-03-14 09:44:27.400333 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | OXSkmmHqQNLCHeKk21bRLWm9D/x02LQe4DBhAhWXi8Fv091sVOyWZyi0aOj4vR72'
2026-03-14 09:44:27.400337 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | zxVKeyC/dk5b8mMpbd8BsUQZymZvx0bICjmCiy3/2XeT6UN+TbByXr1zxreReFX6'
2026-03-14 09:44:27.400342 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | IPTwgxUaLDgMJNckUkXuPWFHpSM9VmgsX7PHF70sBcnohIxcBWQ+mRlLNnvevshr'
2026-03-14 09:44:27.400346 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | tnENhtAWnYo8IzWDuDwGpjlV2udkJm8oAymGjTJZjrLl4cM2/YEPayryilEyCXt6'
2026-03-14 09:44:27.400350 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | dmKKKdNJwwc7daus5jjeB61rj+Euj9i7k4VJ0dbc3hoMawENCUxd1iEpZ8hrYzwd'
2026-03-14 09:44:27.400354 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | gqxSY8FBySOsOpE6f4UCAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l'
2026-03-14 09:44:27.400358 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBRHmG/kNN4R'
2026-03-14 09:44:27.400362 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | AXRuteqRAnTHzA6xPDA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh'
2026-03-14 09:44:27.400367 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBABvVpRyL'
2026-03-14 09:44:27.400377 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | LYOYtFwx705GeRQaYFv7zsVPwEh88v4fVJp78VkDyH3aOHR5MdJUTpSkI+rlAqgn'
2026-03-14 09:44:27.400388 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | FnJo2Q1xCjKLHrk6/306WyHD5YTY9bHAA2fdNDCFI9uUyIe9cp/4+kHSdpcwS5To'
2026-03-14 09:44:27.400404 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | aBjUmgLQrPo3YiUC641+vOJg0dzT4JWi3RqflnWDKYYEHsG1GF/KNLJoTipunDA+'
2026-03-14 09:44:27.400410 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | 6qVljxzBEA2eOfQkRtIcTpbNS2MwYwgnr4IWo6q86d3Lv9/CDP/4JP1q/r2mWBOb'
2026-03-14 09:44:27.400415 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | rTsDohIwyIPV7ZncQZM4mAg7kuyoBeYYxrx67ARSoR0cB0QF3qo3+Ybt78KEeYID'
2026-03-14 09:44:27.400421 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | X1EPIDSyjhO9TLlufzdyXecT3XAjW9PCP8eqGrRlG8qxMhMk4omeVytptmYGTe8r'
2026-03-14 09:44:27.400427 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | YMrn5o11820geCAv39teQAK8VcMAyQSO/S/Ss8Q/XBaY7zonwdMPT6PIRyUfIzAn'
2026-03-14 09:44:27.400432 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | L/XA9s+XEZdGHWLj31RKo7h2C5UQRWhutzugotvFcioF0bVkKTMy/NOf4A=='
2026-03-14 09:44:27.400438 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | -----END
2026-03-14 09:44:27.400444 | controller |       CERTIFICATE-----'' '']'''
2026-03-14 09:44:27.400454 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/1-deploy-with-tlse | test
2026-03-14 09:44:27.400460 | controller |       step completed 1-deploy-with-tlse'
2026-03-14 09:44:27.400466 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400472 | controller |       | starting test step 2-change-public-svc-certificate'
2026-03-14 09:44:27.400478 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400484 | controller |       | running command: [sh -c oc patch Certificate -n $NAMESPACE watcher-public-svc
2026-03-14 09:44:27.400490 | controller |       --type=''json'' -p=''[{"op": "replace", "path": "/spec/dnsNames", "value":[''watcher-public.watcher-kuttl-default.svc'',
2026-03-14 09:44:27.400494 | controller |       ''watcher-public.watcher-kuttl-default.svc.cluster.local'']}]'''
2026-03-14 09:44:27.400498 | controller |     - '        ]'
2026-03-14 09:44:27.400502 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400507 | controller |       | certificate.cert-manager.io/watcher-public-svc patched'
2026-03-14 09:44:27.400511 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400515 | controller |       | running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.400520 | controller |     - '        svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat
2026-03-14 09:44:27.400526 | controller |       /etc/pki/tls/certs/public.crt)'
2026-03-14 09:44:27.400532 | controller |     - '        secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}''
2026-03-14 09:44:27.400538 | controller |       | base64 --decode)'
2026-03-14 09:44:27.400544 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.400550 | controller |     - '        if [ "${svc_cert}" != "${secret_cert}" ]; then'
2026-03-14 09:44:27.400556 | controller |     - '          exit 1'
2026-03-14 09:44:27.400562 | controller |     - '        fi'
2026-03-14 09:44:27.400567 | controller |     - '        ]'
2026-03-14 09:44:27.400573 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400577 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt'
2026-03-14 09:44:27.400581 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400585 | controller |       | + svc_cert=''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.400590 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400594 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.400598 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400602 | controller |       | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.400606 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400611 | controller |       | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.400615 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400619 | controller |       | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.400623 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400627 | controller |       | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.400631 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400636 | controller |       | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.400640 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400653 | controller |       | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.400661 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400665 | controller |       | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.400669 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400674 | controller |       | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.400678 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400682 | controller |       | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.400686 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400690 | controller |       | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.400694 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400699 | controller |       | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.400703 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400707 | controller |       | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.400711 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400715 | controller |       | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.400719 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400723 | controller |       | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.400727 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400732 | controller |       | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.400739 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400743 | controller |       | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.400747 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400751 | controller |       | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.400756 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400760 | controller |       | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.400764 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400768 | controller |       | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.400774 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400778 | controller |       | -----END CERTIFICATE-----'''
2026-03-14 09:44:27.400782 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400786 | controller |       | ++ base64 --decode'
2026-03-14 09:44:27.400790 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400795 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:44:27.400799 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400803 | controller |       | + secret_cert=''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.400807 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400811 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.400815 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400821 | controller |       | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.400825 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400830 | controller |       | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.400834 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400838 | controller |       | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.400842 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400846 | controller |       | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.400850 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400854 | controller |       | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.400859 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400866 | controller |       | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.400870 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400879 | controller |       | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.400884 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400888 | controller |       | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.400892 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400896 | controller |       | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.400900 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400904 | controller |       | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.400909 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400913 | controller |       | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.400917 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400921 | controller |       | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.400925 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400929 | controller |       | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.400933 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400938 | controller |       | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.400942 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400946 | controller |       | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.400950 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400954 | controller |       | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.400958 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400962 | controller |       | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.400966 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400971 | controller |       | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.400975 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400979 | controller |       | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.400985 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400989 | controller |       | -----END CERTIFICATE-----'''
2026-03-14 09:44:27.400993 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.400997 | controller |       | + ''['' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.401001 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401006 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.401011 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401016 | controller |       | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.401020 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401024 | controller |       | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.401028 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401032 | controller |       | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.401037 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401041 | controller |       | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.401045 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401049 | controller |       | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.401053 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401057 | controller |       | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.401062 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401066 | controller |       | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.401070 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401075 | controller |       | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.401079 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401084 | controller |       | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.401088 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401097 | controller |       | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.401101 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401106 | controller |       | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.401110 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401114 | controller |       | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.401118 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401122 | controller |       | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.401126 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401131 | controller |       | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.401135 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401139 | controller |       | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.401143 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401149 | controller |       | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.401153 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401157 | controller |       | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.401162 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401166 | controller |       | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.401170 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401174 | controller |       | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.401178 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401182 | controller |       | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.401187 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401191 | controller |       | MIIDszCCAhugAwIBAgIRAOR/0Fiub301knirimnM4xgwDQYJKoZIhvcNAQEMBQAw'
2026-03-14 09:44:27.401195 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401199 | controller |       | GDEWMBQGA1UEAxMNcm9vdGNhLXB1YmxpYzAeFw0yNjAzMTQwOTM2MjVaFw0zMTAz'
2026-03-14 09:44:27.401203 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401207 | controller |       | MTMwOTM2MjVaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrq6sN'
2026-03-14 09:44:27.401212 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401218 | controller |       | JvN1y3jjPR3eKuVzZBO67xOeDFD+jOdHaRhHHq8pqwT8yv4jLM+lt+iag68uOduz'
2026-03-14 09:44:27.401222 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401227 | controller |       | ycu0r7B/zHUjXBf9jdDyzwQEZSgKRvgAceu2Ig7Lbv7p1C/Mvh0YYM1n1wxJ1onn'
2026-03-14 09:44:27.401231 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401235 | controller |       | MUBL+dp6zUlXfXUenNWcghNw/X9fCkEBNKs+GO6lXo123/5ta8FsGMYnAPRy2Tsk'
2026-03-14 09:44:27.401239 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401243 | controller |       | YNf3w0aEIf+9/rnho7Rvm2Cy2IiJCwE3muyWY0mqGpl93vUJohy1dghZUOQMoMou'
2026-03-14 09:44:27.401248 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401252 | controller |       | F+NAu265JrRRn9torEA1s4HTuNQ2nl6ltBmr6BJ+Gm35REdt7jsNVdAExcCQZF0a'
2026-03-14 09:44:27.401256 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401260 | controller |       | N7yUiCyFGxdcdm/1AgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM'
2026-03-14 09:44:27.401264 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401268 | controller |       | MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUKeRpCtOZVXu7'
2026-03-14 09:44:27.401273 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401277 | controller |       | Ofg9u+EmlddjTzQwNgYDVR0RAQH/BCwwKoIod2F0Y2hlci1wdWJsaWMud2F0Y2hl'
2026-03-14 09:44:27.401308 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401317 | controller |       | ci1rdXR0bC1kZWZhdWx0LnN2YzANBgkqhkiG9w0BAQwFAAOCAYEAb2uWx8KPvqhR'
2026-03-14 09:44:27.401322 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401326 | controller |       | Vks/GsGf38V6lIa0lygt15sYJdO0ara8t/F3PT0Tc5+VoX4Vb0RsUP3Os4q/RJIG'
2026-03-14 09:44:27.401330 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401336 | controller |       | aqspYeNKVUZW3lXXzhLz5oaL0JTan7yrNyBNNGmJ+CJlFNrugFsq+5I+5Hq/DP7J'
2026-03-14 09:44:27.401346 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401351 | controller |       | qZ2n8rIo9OZy4XgAgdbCDO1RImmHXbhOsk3bsnwe9/+s/dD6alZZ3lbAUMaC9mLX'
2026-03-14 09:44:27.401355 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401359 | controller |       | lklzzj5Ypvj8Us67txgDWCc24RpYzywNTyfeGcCASw82kM74pso68JWMhwKx3j6z'
2026-03-14 09:44:27.401364 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401368 | controller |       | FkrNrGe8nhQnhGYYvIpNhXBMcXP0Cjc3R5nkj8c7UsLWQtPJzCO1z29haJp0zCmc'
2026-03-14 09:44:27.401372 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401376 | controller |       | 52bVj1jOJ/oj0w+Cc0z5mu25h3m8+xpoMMRHsYH1cIw66XUHVtQCe43C/qMrpjlG'
2026-03-14 09:44:27.401380 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401384 | controller |       | 5DlgYkaHtJ5gmHxfHvfwSGz2nQWprx8L1226RKNz2L36csCxFJ8W8UWcmUVFYAwK'
2026-03-14 09:44:27.401388 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401393 | controller |       | UIxCOGbSy+tECJeyy0ltN2krgrvayP+PGo6m3o5Q74tWubu9w98A'
2026-03-14 09:44:27.401397 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401401 | controller |       | -----END CERTIFICATE-----'' '']'''
2026-03-14 09:44:27.401405 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/2-change-public-svc-certificate
2026-03-14 09:44:27.401409 | controller |       | test step completed 2-change-public-svc-certificate'
2026-03-14 09:44:27.401413 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401418 | controller |       | starting test step 3-change-internal-svc-certificate'
2026-03-14 09:44:27.401422 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401426 | controller |       | running command: [sh -c oc patch Certificate -n $NAMESPACE watcher-internal-svc
2026-03-14 09:44:27.401430 | controller |       --type=''json'' -p=''[{"op": "replace", "path": "/spec/dnsNames", "value":[''watcher-internal.watcher-kuttl-default.svc'',
2026-03-14 09:44:27.401434 | controller |       ''watcher-internal.watcher-kuttl-default.svc.cluster.local'']}]'''
2026-03-14 09:44:27.401439 | controller |     - '        ]'
2026-03-14 09:44:27.401443 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401447 | controller |       | certificate.cert-manager.io/watcher-internal-svc patched'
2026-03-14 09:44:27.401451 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401455 | controller |       | running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.401459 | controller |     - '        svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat
2026-03-14 09:44:27.401464 | controller |       /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.401468 | controller |     - '        secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o
2026-03-14 09:44:27.401472 | controller |       jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.401476 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.401480 | controller |     - '        if [ "${svc_cert}" != "${secret_cert}" ]; then'
2026-03-14 09:44:27.401485 | controller |     - '          exit 1'
2026-03-14 09:44:27.401489 | controller |     - '        fi'
2026-03-14 09:44:27.401493 | controller |     - '        ]'
2026-03-14 09:44:27.401497 | controller |     - '    logger.go:42: 09:36:49 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401501 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:44:27.401506 | controller |     - '    logger.go:42: 09:36:50 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401511 | controller |       | error: Internal error occurred: error executing command in container: container
2026-03-14 09:44:27.401516 | controller |       is not created or running'
2026-03-14 09:44:27.401520 | controller |     - '    logger.go:42: 09:36:50 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401524 | controller |       | + svc_cert='
2026-03-14 09:44:27.401528 | controller |     - '    logger.go:42: 09:36:51 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401532 | controller |       | running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.401537 | controller |     - '        svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat
2026-03-14 09:44:27.401541 | controller |       /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.401545 | controller |     - '        secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o
2026-03-14 09:44:27.401549 | controller |       jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.401553 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.401558 | controller |     - '        if [ "${svc_cert}" != "${secret_cert}" ]; then'
2026-03-14 09:44:27.401562 | controller |     - '          exit 1'
2026-03-14 09:44:27.401566 | controller |     - '        fi'
2026-03-14 09:44:27.401570 | controller |     - '        ]'
2026-03-14 09:44:27.401574 | controller |     - '    logger.go:42: 09:36:51 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401579 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:44:27.401583 | controller |     - '    logger.go:42: 09:36:51 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401587 | controller |       | error: unable to upgrade connection: container not found ("watcher-api")'
2026-03-14 09:44:27.401597 | controller |     - '    logger.go:42: 09:36:51 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401601 | controller |       | + svc_cert='
2026-03-14 09:44:27.401605 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401610 | controller |       | running command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.401614 | controller |     - '        svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat
2026-03-14 09:44:27.401618 | controller |       /etc/pki/tls/certs/internal.crt)'
2026-03-14 09:44:27.401625 | controller |     - '        secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o
2026-03-14 09:44:27.401629 | controller |       jsonpath=''{.data.tls\.crt}'' | base64 --decode)'
2026-03-14 09:44:27.401633 | controller |     - '        # ensure that the svc secret and cert secret match'
2026-03-14 09:44:27.401637 | controller |     - '        if [ "${svc_cert}" != "${secret_cert}" ]; then'
2026-03-14 09:44:27.401642 | controller |     - '          exit 1'
2026-03-14 09:44:27.401646 | controller |     - '        fi'
2026-03-14 09:44:27.401650 | controller |     - '        ]'
2026-03-14 09:44:27.401654 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401658 | controller |       | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt'
2026-03-14 09:44:27.401663 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401667 | controller |       | + svc_cert=''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.401671 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401675 | controller |       | MIID8DCCAligAwIBAgIQajyja/okveMym9ZKBf72LjANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:44:27.401679 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401684 | controller |       | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDkzNjUwWhcNMzEw'
2026-03-14 09:44:27.401688 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401692 | controller |       | MzEzMDkzNjUwWjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4S4p'
2026-03-14 09:44:27.401698 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401702 | controller |       | yP8aQfvOUpKYA2csCMYtnRCvHUQjFn5L/0uId8KEre8xVSzRZbeTV05hkxlklYaZ'
2026-03-14 09:44:27.401706 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401710 | controller |       | zh79ln9aDCOT6kanAVAppwiA7Gmyw3cLC5SnLT1nAHipvF3r7DqzI7MFgO5gRFgW'
2026-03-14 09:44:27.401714 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401719 | controller |       | HpAoSIsnBPwXt4hnmrw2Vtzdw34ig+letfnOxuZWaU4h8krF+8hCGJ/RLq99ajJy'
2026-03-14 09:44:27.401723 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401727 | controller |       | G6CFAwh3YjWCoqeemDvIW92p2Up9uTyT7Bxzmbe+ybEFSXQAUYzQde3B3mWIBLnz'
2026-03-14 09:44:27.401731 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401735 | controller |       | esOz/0G3/3e24Aj9+7zpcbd0pVuVMUXCV9pEcPo1kTKX+KBgvS6Z+oi7lkeHPXYw'
2026-03-14 09:44:27.401739 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401744 | controller |       | O/lURQupvVThqDbWDwIDAQABo4HLMIHIMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:44:27.401748 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401752 | controller |       | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFEeYb+Q03hEB'
2026-03-14 09:44:27.401756 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401760 | controller |       | dG616pECdMfMDrE8MHIGA1UdEQEB/wRoMGaCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:44:27.401765 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401769 | controller |       | Y2hlci1rdXR0bC1kZWZhdWx0LnN2Y4I4d2F0Y2hlci1pbnRlcm5hbC53YXRjaGVy'
2026-03-14 09:44:27.401773 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401777 | controller |       | LWt1dHRsLWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQAD'
2026-03-14 09:44:27.401781 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401785 | controller |       | ggGBAK6NF533YWWeQT84GLLzJ0ogHXKIDKZVRwqLaNXLWGG0W03Cow9vch6+Fv0b'
2026-03-14 09:44:27.401790 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401794 | controller |       | rkJisw8FbyD772PLovRl0hsUOW2DQw+R7qtNsCHUTxLMDnBkJ1QBYPJS80LumsGR'
2026-03-14 09:44:27.401798 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401804 | controller |       | vVXFLl69tqGXkoaOyYgogD7HHJO69quA7qENirYSK1RHdmoqF5CX2qKJnLdnEgmP'
2026-03-14 09:44:27.401808 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401812 | controller |       | ZFGE1JshS1XaVoz4DYDTlqq8FHZ0/Jigrj8EG1Rjpv6+qA8m2ZEsk8ISyFmzwWqi'
2026-03-14 09:44:27.401816 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401821 | controller |       | FGw1GV874NeqiB3ip4atDzH2ossazs4WNn65dA757uk6lICb2vKbXzNTadZRTQSM'
2026-03-14 09:44:27.401830 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401835 | controller |       | FehTsYpmDx5lc1y9K1MpBOmZmGRXIfHkrg2YVRsFq72bZQlRfop1dOgXDKRImakm'
2026-03-14 09:44:27.401839 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401843 | controller |       | F9+dzFNDrHO6H3MHI6bnC2DQUWI2FOvAJ8L93wedHbyR8sJ8ObnJli7pngCIY1Rp'
2026-03-14 09:44:27.401848 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401852 | controller |       | E8tkXhzRn5kiy7dXmeP4LYghoCN/8q0zIct6CNUFL3VMCCm9EWruWCiFr+G60Nz0'
2026-03-14 09:44:27.401858 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401862 | controller |       | G8kFPA=='
2026-03-14 09:44:27.401866 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401871 | controller |       | -----END CERTIFICATE-----'''
2026-03-14 09:44:27.401875 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401879 | controller |       | ++ base64 --decode'
2026-03-14 09:44:27.401883 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401887 | controller |       | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}'''
2026-03-14 09:44:27.401892 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401896 | controller |       | + secret_cert=''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.401900 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401904 | controller |       | MIID8DCCAligAwIBAgIQajyja/okveMym9ZKBf72LjANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:44:27.401908 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401913 | controller |       | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDkzNjUwWhcNMzEw'
2026-03-14 09:44:27.401918 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401922 | controller |       | MzEzMDkzNjUwWjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4S4p'
2026-03-14 09:44:27.401927 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401931 | controller |       | yP8aQfvOUpKYA2csCMYtnRCvHUQjFn5L/0uId8KEre8xVSzRZbeTV05hkxlklYaZ'
2026-03-14 09:44:27.401935 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401939 | controller |       | zh79ln9aDCOT6kanAVAppwiA7Gmyw3cLC5SnLT1nAHipvF3r7DqzI7MFgO5gRFgW'
2026-03-14 09:44:27.401943 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401948 | controller |       | HpAoSIsnBPwXt4hnmrw2Vtzdw34ig+letfnOxuZWaU4h8krF+8hCGJ/RLq99ajJy'
2026-03-14 09:44:27.401952 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401956 | controller |       | G6CFAwh3YjWCoqeemDvIW92p2Up9uTyT7Bxzmbe+ybEFSXQAUYzQde3B3mWIBLnz'
2026-03-14 09:44:27.401960 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401964 | controller |       | esOz/0G3/3e24Aj9+7zpcbd0pVuVMUXCV9pEcPo1kTKX+KBgvS6Z+oi7lkeHPXYw'
2026-03-14 09:44:27.401968 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401972 | controller |       | O/lURQupvVThqDbWDwIDAQABo4HLMIHIMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:44:27.401977 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401981 | controller |       | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFEeYb+Q03hEB'
2026-03-14 09:44:27.401985 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401989 | controller |       | dG616pECdMfMDrE8MHIGA1UdEQEB/wRoMGaCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:44:27.401993 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.401997 | controller |       | Y2hlci1rdXR0bC1kZWZhdWx0LnN2Y4I4d2F0Y2hlci1pbnRlcm5hbC53YXRjaGVy'
2026-03-14 09:44:27.402002 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402006 | controller |       | LWt1dHRsLWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQAD'
2026-03-14 09:44:27.402010 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402014 | controller |       | ggGBAK6NF533YWWeQT84GLLzJ0ogHXKIDKZVRwqLaNXLWGG0W03Cow9vch6+Fv0b'
2026-03-14 09:44:27.402020 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402024 | controller |       | rkJisw8FbyD772PLovRl0hsUOW2DQw+R7qtNsCHUTxLMDnBkJ1QBYPJS80LumsGR'
2026-03-14 09:44:27.402028 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402033 | controller |       | vVXFLl69tqGXkoaOyYgogD7HHJO69quA7qENirYSK1RHdmoqF5CX2qKJnLdnEgmP'
2026-03-14 09:44:27.402037 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402041 | controller |       | ZFGE1JshS1XaVoz4DYDTlqq8FHZ0/Jigrj8EG1Rjpv6+qA8m2ZEsk8ISyFmzwWqi'
2026-03-14 09:44:27.402050 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402055 | controller |       | FGw1GV874NeqiB3ip4atDzH2ossazs4WNn65dA757uk6lICb2vKbXzNTadZRTQSM'
2026-03-14 09:44:27.402059 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402063 | controller |       | FehTsYpmDx5lc1y9K1MpBOmZmGRXIfHkrg2YVRsFq72bZQlRfop1dOgXDKRImakm'
2026-03-14 09:44:27.402067 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402072 | controller |       | F9+dzFNDrHO6H3MHI6bnC2DQUWI2FOvAJ8L93wedHbyR8sJ8ObnJli7pngCIY1Rp'
2026-03-14 09:44:27.402076 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402080 | controller |       | E8tkXhzRn5kiy7dXmeP4LYghoCN/8q0zIct6CNUFL3VMCCm9EWruWCiFr+G60Nz0'
2026-03-14 09:44:27.402084 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402088 | controller |       | G8kFPA=='
2026-03-14 09:44:27.402093 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402097 | controller |       | -----END CERTIFICATE-----'''
2026-03-14 09:44:27.402101 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402105 | controller |       | + ''['' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.402109 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402113 | controller |       | MIID8DCCAligAwIBAgIQajyja/okveMym9ZKBf72LjANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:44:27.402118 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402122 | controller |       | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDkzNjUwWhcNMzEw'
2026-03-14 09:44:27.402129 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402160 | controller |       | MzEzMDkzNjUwWjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4S4p'
2026-03-14 09:44:27.402165 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402170 | controller |       | yP8aQfvOUpKYA2csCMYtnRCvHUQjFn5L/0uId8KEre8xVSzRZbeTV05hkxlklYaZ'
2026-03-14 09:44:27.402175 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402180 | controller |       | zh79ln9aDCOT6kanAVAppwiA7Gmyw3cLC5SnLT1nAHipvF3r7DqzI7MFgO5gRFgW'
2026-03-14 09:44:27.402185 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402190 | controller |       | HpAoSIsnBPwXt4hnmrw2Vtzdw34ig+letfnOxuZWaU4h8krF+8hCGJ/RLq99ajJy'
2026-03-14 09:44:27.402194 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402199 | controller |       | G6CFAwh3YjWCoqeemDvIW92p2Up9uTyT7Bxzmbe+ybEFSXQAUYzQde3B3mWIBLnz'
2026-03-14 09:44:27.402204 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402209 | controller |       | esOz/0G3/3e24Aj9+7zpcbd0pVuVMUXCV9pEcPo1kTKX+KBgvS6Z+oi7lkeHPXYw'
2026-03-14 09:44:27.402214 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402221 | controller |       | O/lURQupvVThqDbWDwIDAQABo4HLMIHIMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:44:27.402230 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402236 | controller |       | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFEeYb+Q03hEB'
2026-03-14 09:44:27.402241 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402246 | controller |       | dG616pECdMfMDrE8MHIGA1UdEQEB/wRoMGaCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:44:27.402251 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402256 | controller |       | Y2hlci1rdXR0bC1kZWZhdWx0LnN2Y4I4d2F0Y2hlci1pbnRlcm5hbC53YXRjaGVy'
2026-03-14 09:44:27.402261 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402265 | controller |       | LWt1dHRsLWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQAD'
2026-03-14 09:44:27.402270 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402275 | controller |       | ggGBAK6NF533YWWeQT84GLLzJ0ogHXKIDKZVRwqLaNXLWGG0W03Cow9vch6+Fv0b'
2026-03-14 09:44:27.402280 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402285 | controller |       | rkJisw8FbyD772PLovRl0hsUOW2DQw+R7qtNsCHUTxLMDnBkJ1QBYPJS80LumsGR'
2026-03-14 09:44:27.402301 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402308 | controller |       | vVXFLl69tqGXkoaOyYgogD7HHJO69quA7qENirYSK1RHdmoqF5CX2qKJnLdnEgmP'
2026-03-14 09:44:27.402313 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402318 | controller |       | ZFGE1JshS1XaVoz4DYDTlqq8FHZ0/Jigrj8EG1Rjpv6+qA8m2ZEsk8ISyFmzwWqi'
2026-03-14 09:44:27.402323 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402334 | controller |       | FGw1GV874NeqiB3ip4atDzH2ossazs4WNn65dA757uk6lICb2vKbXzNTadZRTQSM'
2026-03-14 09:44:27.402339 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402344 | controller |       | FehTsYpmDx5lc1y9K1MpBOmZmGRXIfHkrg2YVRsFq72bZQlRfop1dOgXDKRImakm'
2026-03-14 09:44:27.402349 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402354 | controller |       | F9+dzFNDrHO6H3MHI6bnC2DQUWI2FOvAJ8L93wedHbyR8sJ8ObnJli7pngCIY1Rp'
2026-03-14 09:44:27.402359 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402364 | controller |       | E8tkXhzRn5kiy7dXmeP4LYghoCN/8q0zIct6CNUFL3VMCCm9EWruWCiFr+G60Nz0'
2026-03-14 09:44:27.402369 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402374 | controller |       | G8kFPA=='
2026-03-14 09:44:27.402379 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402384 | controller |       | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----'
2026-03-14 09:44:27.402388 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402393 | controller |       | MIID8DCCAligAwIBAgIQajyja/okveMym9ZKBf72LjANBgkqhkiG9w0BAQwFADAa'
2026-03-14 09:44:27.402397 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402401 | controller |       | MRgwFgYDVQQDEw9yb290Y2EtaW50ZXJuYWwwHhcNMjYwMzE0MDkzNjUwWhcNMzEw'
2026-03-14 09:44:27.402405 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402409 | controller |       | MzEzMDkzNjUwWjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4S4p'
2026-03-14 09:44:27.402414 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402420 | controller |       | yP8aQfvOUpKYA2csCMYtnRCvHUQjFn5L/0uId8KEre8xVSzRZbeTV05hkxlklYaZ'
2026-03-14 09:44:27.402424 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402428 | controller |       | zh79ln9aDCOT6kanAVAppwiA7Gmyw3cLC5SnLT1nAHipvF3r7DqzI7MFgO5gRFgW'
2026-03-14 09:44:27.402433 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402437 | controller |       | HpAoSIsnBPwXt4hnmrw2Vtzdw34ig+letfnOxuZWaU4h8krF+8hCGJ/RLq99ajJy'
2026-03-14 09:44:27.402441 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402445 | controller |       | G6CFAwh3YjWCoqeemDvIW92p2Up9uTyT7Bxzmbe+ybEFSXQAUYzQde3B3mWIBLnz'
2026-03-14 09:44:27.402449 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402453 | controller |       | esOz/0G3/3e24Aj9+7zpcbd0pVuVMUXCV9pEcPo1kTKX+KBgvS6Z+oi7lkeHPXYw'
2026-03-14 09:44:27.402458 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402462 | controller |       | O/lURQupvVThqDbWDwIDAQABo4HLMIHIMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE'
2026-03-14 09:44:27.402466 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402470 | controller |       | DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFEeYb+Q03hEB'
2026-03-14 09:44:27.402474 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402479 | controller |       | dG616pECdMfMDrE8MHIGA1UdEQEB/wRoMGaCKndhdGNoZXItaW50ZXJuYWwud2F0'
2026-03-14 09:44:27.402483 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402487 | controller |       | Y2hlci1rdXR0bC1kZWZhdWx0LnN2Y4I4d2F0Y2hlci1pbnRlcm5hbC53YXRjaGVy'
2026-03-14 09:44:27.402491 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402495 | controller |       | LWt1dHRsLWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQAD'
2026-03-14 09:44:27.402500 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402504 | controller |       | ggGBAK6NF533YWWeQT84GLLzJ0ogHXKIDKZVRwqLaNXLWGG0W03Cow9vch6+Fv0b'
2026-03-14 09:44:27.402508 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402512 | controller |       | rkJisw8FbyD772PLovRl0hsUOW2DQw+R7qtNsCHUTxLMDnBkJ1QBYPJS80LumsGR'
2026-03-14 09:44:27.402516 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402521 | controller |       | vVXFLl69tqGXkoaOyYgogD7HHJO69quA7qENirYSK1RHdmoqF5CX2qKJnLdnEgmP'
2026-03-14 09:44:27.402525 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402529 | controller |       | ZFGE1JshS1XaVoz4DYDTlqq8FHZ0/Jigrj8EG1Rjpv6+qA8m2ZEsk8ISyFmzwWqi'
2026-03-14 09:44:27.402535 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402539 | controller |       | FGw1GV874NeqiB3ip4atDzH2ossazs4WNn65dA757uk6lICb2vKbXzNTadZRTQSM'
2026-03-14 09:44:27.402543 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402548 | controller |       | FehTsYpmDx5lc1y9K1MpBOmZmGRXIfHkrg2YVRsFq72bZQlRfop1dOgXDKRImakm'
2026-03-14 09:44:27.402557 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402562 | controller |       | F9+dzFNDrHO6H3MHI6bnC2DQUWI2FOvAJ8L93wedHbyR8sJ8ObnJli7pngCIY1Rp'
2026-03-14 09:44:27.402566 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402570 | controller |       | E8tkXhzRn5kiy7dXmeP4LYghoCN/8q0zIct6CNUFL3VMCCm9EWruWCiFr+G60Nz0'
2026-03-14 09:44:27.402575 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402581 | controller |       | G8kFPA=='
2026-03-14 09:44:27.402585 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402589 | controller |       | -----END CERTIFICATE-----'' '']'''
2026-03-14 09:44:27.402593 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/3-change-internal-svc-certificate
2026-03-14 09:44:27.402597 | controller |       | test step completed 3-change-internal-svc-certificate'
2026-03-14 09:44:27.402602 | controller |     - '    logger.go:42: 09:36:52 | watcher-tls-certs-change/4-cleanup-watcher | starting
2026-03-14 09:44:27.402606 | controller |       test step 4-cleanup-watcher'
2026-03-14 09:44:27.402610 | controller |     - '    logger.go:42: 09:37:00 | watcher-tls-certs-change/4-cleanup-watcher | test
2026-03-14 09:44:27.402614 | controller |       step completed 4-cleanup-watcher'
2026-03-14 09:44:27.402618 | controller |     - '    logger.go:42: 09:37:00 | watcher-tls-certs-change/5-clenaup-certs | starting
2026-03-14 09:44:27.402623 | controller |       test step 5-clenaup-certs'
2026-03-14 09:44:27.402627 | controller |     - '    logger.go:42: 09:37:00 | watcher-tls-certs-change/5-clenaup-certs | test step
2026-03-14 09:44:27.402631 | controller |       completed 5-clenaup-certs'
2026-03-14 09:44:27.402635 | controller |     - '    logger.go:42: 09:37:00 | watcher-tls-certs-change | skipping kubernetes event
2026-03-14 09:44:27.402639 | controller |       logging'
2026-03-14 09:44:27.402644 | controller |     - === CONT  kuttl/harness/watcher-tls
2026-03-14 09:44:27.402648 | controller |     - '    logger.go:42: 09:37:00 | watcher-tls | Skipping creation of user-supplied namespace:
2026-03-14 09:44:27.402652 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.402656 | controller |     - '    logger.go:42: 09:37:00 | watcher-tls/0-cleanup-watcher | starting test step
2026-03-14 09:44:27.402660 | controller |       0-cleanup-watcher'
2026-03-14 09:44:27.402665 | controller |     - '    logger.go:42: 09:37:00 | watcher-tls/0-cleanup-watcher | test step completed
2026-03-14 09:44:27.402669 | controller |       0-cleanup-watcher'
2026-03-14 09:44:27.402673 | controller |     - '    logger.go:42: 09:37:00 | watcher-tls/1-deploy-with-tlse | starting test step
2026-03-14 09:44:27.402677 | controller |       1-deploy-with-tlse'
2026-03-14 09:44:27.402681 | controller |     - '    logger.go:42: 09:37:00 | watcher-tls/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-internal-svc
2026-03-14 09:44:27.402686 | controller |       created'
2026-03-14 09:44:27.402690 | controller |     - '    logger.go:42: 09:37:00 | watcher-tls/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc
2026-03-14 09:44:27.402694 | controller |       created'
2026-03-14 09:44:27.402698 | controller |     - '    logger.go:42: 09:37:00 | watcher-tls/1-deploy-with-tlse | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:44:27.402703 | controller |       created'
2026-03-14 09:44:27.402707 | controller |     - '    logger.go:42: 09:37:00 | watcher-tls/1-deploy-with-tlse | running command:
2026-03-14 09:44:27.402711 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.402715 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.402719 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.402724 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.402728 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.402732 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.402736 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.402740 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.402744 | controller |       ]'
2026-03-14 09:44:27.402749 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64
2026-03-14 09:44:27.402753 | controller |       -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:44:27.402757 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64
2026-03-14 09:44:27.402761 | controller |       -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2
2026-03-14 09:44:27.402767 | controller |       ]'
2026-03-14 09:44:27.402772 | controller |     - '        # check that both endpoints have https set'
2026-03-14 09:44:27.402776 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.402780 | controller |       infra-optim | [ $(grep -c https) == 2 ]'
2026-03-14 09:44:27.402784 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.402789 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.402793 | controller |       == "" ]; then'
2026-03-14 09:44:27.402797 | controller |     - '            exit 0'
2026-03-14 09:44:27.402801 | controller |     - '        fi'
2026-03-14 09:44:27.402805 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.402811 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.402815 | controller |     - '        counter=0'
2026-03-14 09:44:27.402820 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.402824 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.402828 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.402832 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.402836 | controller |     - '            fi'
2026-03-14 09:44:27.402845 | controller |     - '        done'
2026-03-14 09:44:27.402851 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.402855 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.402859 | controller |     - '          exit 1'
2026-03-14 09:44:27.402863 | controller |     - '        else'
2026-03-14 09:44:27.402868 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.402872 | controller |     - '        fi'
2026-03-14 09:44:27.402876 | controller |     - '        ]'
2026-03-14 09:44:27.402880 | controller |     - '    logger.go:42: 09:37:00 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.402884 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.402888 | controller |     - '    logger.go:42: 09:37:00 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.402893 | controller |     - '    logger.go:42: 09:37:03 | watcher-tls/1-deploy-with-tlse | + ''['' 0 == 1 '']'''
2026-03-14 09:44:27.402897 | controller |     - '    logger.go:42: 09:37:04 | watcher-tls/1-deploy-with-tlse | running command:
2026-03-14 09:44:27.402901 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.402905 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.402909 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.402914 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.402918 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.402922 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.402926 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.402930 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.402935 | controller |       ]'
2026-03-14 09:44:27.402939 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64
2026-03-14 09:44:27.402943 | controller |       -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:44:27.402947 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64
2026-03-14 09:44:27.402951 | controller |       -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2
2026-03-14 09:44:27.402955 | controller |       ]'
2026-03-14 09:44:27.402960 | controller |     - '        # check that both endpoints have https set'
2026-03-14 09:44:27.402964 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.402970 | controller |       infra-optim | [ $(grep -c https) == 2 ]'
2026-03-14 09:44:27.402974 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.402978 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.402982 | controller |       == "" ]; then'
2026-03-14 09:44:27.402987 | controller |     - '            exit 0'
2026-03-14 09:44:27.402993 | controller |     - '        fi'
2026-03-14 09:44:27.402997 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.403001 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.403005 | controller |     - '        counter=0'
2026-03-14 09:44:27.403010 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.403014 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.403018 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.403022 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.403026 | controller |     - '            fi'
2026-03-14 09:44:27.403031 | controller |     - '        done'
2026-03-14 09:44:27.403035 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.403039 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.403043 | controller |     - '          exit 1'
2026-03-14 09:44:27.403048 | controller |     - '        else'
2026-03-14 09:44:27.403052 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.403056 | controller |     - '        fi'
2026-03-14 09:44:27.403060 | controller |     - '        ]'
2026-03-14 09:44:27.403064 | controller |     - '    logger.go:42: 09:37:04 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.403068 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.403073 | controller |     - '    logger.go:42: 09:37:04 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.403077 | controller |     - '    logger.go:42: 09:37:06 | watcher-tls/1-deploy-with-tlse | + ''['' 0 == 1 '']'''
2026-03-14 09:44:27.403083 | controller |     - '    logger.go:42: 09:37:08 | watcher-tls/1-deploy-with-tlse | running command:
2026-03-14 09:44:27.403087 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.403091 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.403095 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.403100 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.403104 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.403108 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.403112 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.403119 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.403123 | controller |       ]'
2026-03-14 09:44:27.403129 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64
2026-03-14 09:44:27.403133 | controller |       -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:44:27.403142 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64
2026-03-14 09:44:27.403148 | controller |       -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2
2026-03-14 09:44:27.403152 | controller |       ]'
2026-03-14 09:44:27.403156 | controller |     - '        # check that both endpoints have https set'
2026-03-14 09:44:27.403160 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.403164 | controller |       infra-optim | [ $(grep -c https) == 2 ]'
2026-03-14 09:44:27.403169 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.403173 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.403179 | controller |       == "" ]; then'
2026-03-14 09:44:27.403183 | controller |     - '            exit 0'
2026-03-14 09:44:27.403187 | controller |     - '        fi'
2026-03-14 09:44:27.403192 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.403196 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.403200 | controller |     - '        counter=0'
2026-03-14 09:44:27.403204 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.403208 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.403212 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.403217 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.403221 | controller |     - '            fi'
2026-03-14 09:44:27.403225 | controller |     - '        done'
2026-03-14 09:44:27.403229 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.403234 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.403238 | controller |     - '          exit 1'
2026-03-14 09:44:27.403242 | controller |     - '        else'
2026-03-14 09:44:27.403246 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.403250 | controller |     - '        fi'
2026-03-14 09:44:27.403255 | controller |     - '        ]'
2026-03-14 09:44:27.403259 | controller |     - '    logger.go:42: 09:37:08 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.403263 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.403267 | controller |     - '    logger.go:42: 09:37:08 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.403271 | controller |     - '    logger.go:42: 09:37:10 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.403276 | controller |     - '    logger.go:42: 09:37:10 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.403280 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:44:27.403284 | controller |     - '    logger.go:42: 09:37:10 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}'''
2026-03-14 09:44:27.403303 | controller |     - '    logger.go:42: 09:37:10 | watcher-tls/1-deploy-with-tlse | ++ grep watcher'
2026-03-14 09:44:27.403308 | controller |     - '    logger.go:42: 09:37:12 | watcher-tls/1-deploy-with-tlse | + SERVICEID=e0a908c724554093bb8de86515131a2c'
2026-03-14 09:44:27.403312 | controller |     - '    logger.go:42: 09:37:12 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.403316 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.403323 | controller |     - '    logger.go:42: 09:37:12 | watcher-tls/1-deploy-with-tlse | + ''['' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.403327 | controller |       == e0a908c724554093bb8de86515131a2c '']'''
2026-03-14 09:44:27.403331 | controller |     - '    logger.go:42: 09:37:12 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.403336 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.403340 | controller |     - '    logger.go:42: 09:37:12 | watcher-tls/1-deploy-with-tlse | + ''['' -n '''' '']'''
2026-03-14 09:44:27.403345 | controller |     - '    logger.go:42: 09:37:13 | watcher-tls/1-deploy-with-tlse | running command:
2026-03-14 09:44:27.403349 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.403355 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.403359 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.403363 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.403368 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.403372 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.403376 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.403380 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.403388 | controller |       ]'
2026-03-14 09:44:27.403392 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64
2026-03-14 09:44:27.403396 | controller |       -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:44:27.403401 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64
2026-03-14 09:44:27.403405 | controller |       -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2
2026-03-14 09:44:27.403409 | controller |       ]'
2026-03-14 09:44:27.403413 | controller |     - '        # check that both endpoints have https set'
2026-03-14 09:44:27.403418 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.403422 | controller |       infra-optim | [ $(grep -c https) == 2 ]'
2026-03-14 09:44:27.403426 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.403436 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.403442 | controller |       == "" ]; then'
2026-03-14 09:44:27.403446 | controller |     - '            exit 0'
2026-03-14 09:44:27.403450 | controller |     - '        fi'
2026-03-14 09:44:27.403454 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.403461 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.403465 | controller |     - '        counter=0'
2026-03-14 09:44:27.403469 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.403473 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.403477 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.403482 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.403486 | controller |     - '            fi'
2026-03-14 09:44:27.403490 | controller |     - '        done'
2026-03-14 09:44:27.403494 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.403498 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.403503 | controller |     - '          exit 1'
2026-03-14 09:44:27.403507 | controller |     - '        else'
2026-03-14 09:44:27.403511 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.403515 | controller |     - '        fi'
2026-03-14 09:44:27.403519 | controller |     - '        ]'
2026-03-14 09:44:27.403524 | controller |     - '    logger.go:42: 09:37:13 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.403528 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.403532 | controller |     - '    logger.go:42: 09:37:13 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.403536 | controller |     - '    logger.go:42: 09:37:15 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.403540 | controller |     - '    logger.go:42: 09:37:15 | watcher-tls/1-deploy-with-tlse | ++ grep watcher'
2026-03-14 09:44:27.403545 | controller |     - '    logger.go:42: 09:37:15 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.403549 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:44:27.403553 | controller |     - '    logger.go:42: 09:37:15 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}'''
2026-03-14 09:44:27.403557 | controller |     - '    logger.go:42: 09:37:17 | watcher-tls/1-deploy-with-tlse | + SERVICEID=e0a908c724554093bb8de86515131a2c'
2026-03-14 09:44:27.403561 | controller |     - '    logger.go:42: 09:37:17 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.403566 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.403570 | controller |     - '    logger.go:42: 09:37:17 | watcher-tls/1-deploy-with-tlse | + ''['' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.403574 | controller |       == e0a908c724554093bb8de86515131a2c '']'''
2026-03-14 09:44:27.403580 | controller |     - '    logger.go:42: 09:37:17 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.403584 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.403589 | controller |     - '    logger.go:42: 09:37:18 | watcher-tls/1-deploy-with-tlse | + ''['' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.403593 | controller |       '']'''
2026-03-14 09:44:27.403597 | controller |     - '    logger.go:42: 09:37:18 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.403601 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}'''
2026-03-14 09:44:27.403606 | controller |     - '    logger.go:42: 09:37:18 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1'
2026-03-14 09:44:27.403610 | controller |     - '    logger.go:42: 09:37:18 | watcher-tls/1-deploy-with-tlse | ++ base64 -d'
2026-03-14 09:44:27.403614 | controller |     - '    logger.go:42: 09:37:18 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.403618 | controller |     - '    logger.go:42: 09:37:18 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.403622 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:44:27.403627 | controller |     - '    logger.go:42: 09:37:18 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''cafile
2026-03-14 09:44:27.403631 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'''
2026-03-14 09:44:27.403635 | controller |     - '    logger.go:42: 09:37:18 | watcher-tls/1-deploy-with-tlse | ++ base64 -d'
2026-03-14 09:44:27.403639 | controller |     - '    logger.go:42: 09:37:18 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']'''
2026-03-14 09:44:27.403643 | controller |     - '    logger.go:42: 09:37:18 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.403648 | controller |       openstackclient -- openstack endpoint list'
2026-03-14 09:44:27.403652 | controller |     - '    logger.go:42: 09:37:18 | watcher-tls/1-deploy-with-tlse | + grep infra-optim'
2026-03-14 09:44:27.403657 | controller |     - '    logger.go:42: 09:37:18 | watcher-tls/1-deploy-with-tlse | ++ grep -c https'
2026-03-14 09:44:27.403661 | controller |     - '    logger.go:42: 09:37:20 | watcher-tls/1-deploy-with-tlse | + ''['' 0 == 2 '']'''
2026-03-14 09:44:27.403666 | controller |     - '    logger.go:42: 09:37:21 | watcher-tls/1-deploy-with-tlse | running command:
2026-03-14 09:44:27.403670 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.403674 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.403678 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.403688 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.403692 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.403697 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.403701 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.403705 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.403709 | controller |       ]'
2026-03-14 09:44:27.403714 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64
2026-03-14 09:44:27.403718 | controller |       -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:44:27.403722 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64
2026-03-14 09:44:27.403726 | controller |       -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2
2026-03-14 09:44:27.403730 | controller |       ]'
2026-03-14 09:44:27.403735 | controller |     - '        # check that both endpoints have https set'
2026-03-14 09:44:27.403739 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.403743 | controller |       infra-optim | [ $(grep -c https) == 2 ]'
2026-03-14 09:44:27.403749 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.403753 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.403757 | controller |       == "" ]; then'
2026-03-14 09:44:27.403762 | controller |     - '            exit 0'
2026-03-14 09:44:27.403766 | controller |     - '        fi'
2026-03-14 09:44:27.403770 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.403774 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.403779 | controller |     - '        counter=0'
2026-03-14 09:44:27.403783 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.403787 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.403791 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.403795 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.403800 | controller |     - '            fi'
2026-03-14 09:44:27.403804 | controller |     - '        done'
2026-03-14 09:44:27.403808 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.403812 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.403817 | controller |     - '          exit 1'
2026-03-14 09:44:27.403821 | controller |     - '        else'
2026-03-14 09:44:27.403827 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.403831 | controller |     - '        fi'
2026-03-14 09:44:27.403835 | controller |     - '        ]'
2026-03-14 09:44:27.403839 | controller |     - '    logger.go:42: 09:37:21 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.403844 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.403848 | controller |     - '    logger.go:42: 09:37:21 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.403852 | controller |     - '    logger.go:42: 09:37:23 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.403856 | controller |     - '    logger.go:42: 09:37:23 | watcher-tls/1-deploy-with-tlse | ++ grep watcher'
2026-03-14 09:44:27.403860 | controller |     - '    logger.go:42: 09:37:23 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}'''
2026-03-14 09:44:27.403865 | controller |     - '    logger.go:42: 09:37:23 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.403869 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:44:27.403873 | controller |     - '    logger.go:42: 09:37:25 | watcher-tls/1-deploy-with-tlse | + SERVICEID=e0a908c724554093bb8de86515131a2c'
2026-03-14 09:44:27.403877 | controller |     - '    logger.go:42: 09:37:25 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.403882 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.403886 | controller |     - '    logger.go:42: 09:37:25 | watcher-tls/1-deploy-with-tlse | + ''['' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.403891 | controller |       == e0a908c724554093bb8de86515131a2c '']'''
2026-03-14 09:44:27.403895 | controller |     - '    logger.go:42: 09:37:25 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.403899 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.403904 | controller |     - '    logger.go:42: 09:37:25 | watcher-tls/1-deploy-with-tlse | + ''['' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.403908 | controller |       '']'''
2026-03-14 09:44:27.403912 | controller |     - '    logger.go:42: 09:37:25 | watcher-tls/1-deploy-with-tlse | ++ base64 -d'
2026-03-14 09:44:27.403916 | controller |     - '    logger.go:42: 09:37:25 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.403920 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}'''
2026-03-14 09:44:27.403925 | controller |     - '    logger.go:42: 09:37:25 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1'
2026-03-14 09:44:27.403929 | controller |     - '    logger.go:42: 09:37:26 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.403934 | controller |     - '    logger.go:42: 09:37:26 | watcher-tls/1-deploy-with-tlse | ++ base64 -d'
2026-03-14 09:44:27.403944 | controller |     - '    logger.go:42: 09:37:26 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.403948 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:44:27.403953 | controller |     - '    logger.go:42: 09:37:26 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''cafile
2026-03-14 09:44:27.403957 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'''
2026-03-14 09:44:27.403961 | controller |     - '    logger.go:42: 09:37:26 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']'''
2026-03-14 09:44:27.403965 | controller |     - '    logger.go:42: 09:37:26 | watcher-tls/1-deploy-with-tlse | + grep infra-optim'
2026-03-14 09:44:27.403969 | controller |     - '    logger.go:42: 09:37:26 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.403973 | controller |       openstackclient -- openstack endpoint list'
2026-03-14 09:44:27.403978 | controller |     - '    logger.go:42: 09:37:26 | watcher-tls/1-deploy-with-tlse | ++ grep -c https'
2026-03-14 09:44:27.403982 | controller |     - '    logger.go:42: 09:37:28 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']'''
2026-03-14 09:44:27.403986 | controller |     - '    logger.go:42: 09:37:28 | watcher-tls/1-deploy-with-tlse | ++ oc get pods -n
2026-03-14 09:44:27.403990 | controller |       openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:44:27.403994 | controller |     - '    logger.go:42: 09:37:28 | watcher-tls/1-deploy-with-tlse | + ''['' '''' == ''''
2026-03-14 09:44:27.403998 | controller |       '']'''
2026-03-14 09:44:27.404003 | controller |     - '    logger.go:42: 09:37:28 | watcher-tls/1-deploy-with-tlse | + exit 0'
2026-03-14 09:44:27.404007 | controller |     - '    logger.go:42: 09:37:29 | watcher-tls/1-deploy-with-tlse | running command:
2026-03-14 09:44:27.404011 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.404015 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.404019 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.404024 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.404028 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.404032 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.404036 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.404040 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.404044 | controller |       ]'
2026-03-14 09:44:27.404049 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64
2026-03-14 09:44:27.404053 | controller |       -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:44:27.404057 | controller |     - '        [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64
2026-03-14 09:44:27.404061 | controller |       -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2
2026-03-14 09:44:27.404065 | controller |       ]'
2026-03-14 09:44:27.404070 | controller |     - '        # check that both endpoints have https set'
2026-03-14 09:44:27.404074 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.404078 | controller |       infra-optim | [ $(grep -c https) == 2 ]'
2026-03-14 09:44:27.404082 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.404087 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.404091 | controller |       == "" ]; then'
2026-03-14 09:44:27.404095 | controller |     - '            exit 0'
2026-03-14 09:44:27.404099 | controller |     - '        fi'
2026-03-14 09:44:27.404103 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.404109 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.404113 | controller |     - '        counter=0'
2026-03-14 09:44:27.404118 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.404122 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.404126 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.404130 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.404134 | controller |     - '            fi'
2026-03-14 09:44:27.404139 | controller |     - '        done'
2026-03-14 09:44:27.404143 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.404147 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.404151 | controller |     - '          exit 1'
2026-03-14 09:44:27.404155 | controller |     - '        else'
2026-03-14 09:44:27.404160 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.404164 | controller |     - '        fi'
2026-03-14 09:44:27.404168 | controller |     - '        ]'
2026-03-14 09:44:27.404172 | controller |     - '    logger.go:42: 09:37:29 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.404176 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.404183 | controller |     - '    logger.go:42: 09:37:29 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.404187 | controller |     - '    logger.go:42: 09:37:31 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.404192 | controller |     - '    logger.go:42: 09:37:31 | watcher-tls/1-deploy-with-tlse | ++ grep watcher'
2026-03-14 09:44:27.404196 | controller |     - '    logger.go:42: 09:37:31 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}'''
2026-03-14 09:44:27.404201 | controller |     - '    logger.go:42: 09:37:31 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.404206 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:44:27.404215 | controller |     - '    logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse | + SERVICEID=e0a908c724554093bb8de86515131a2c'
2026-03-14 09:44:27.404220 | controller |     - '    logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.404224 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.404228 | controller |     - '    logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse | + ''['' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.404233 | controller |       == e0a908c724554093bb8de86515131a2c '']'''
2026-03-14 09:44:27.404237 | controller |     - '    logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.404241 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.404245 | controller |     - '    logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse | + ''['' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.404249 | controller |       '']'''
2026-03-14 09:44:27.404254 | controller |     - '    logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse | ++ base64 -d'
2026-03-14 09:44:27.404258 | controller |     - '    logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.404262 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}'''
2026-03-14 09:44:27.404266 | controller |     - '    logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1'
2026-03-14 09:44:27.404270 | controller |     - '    logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.404275 | controller |     - '    logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse | ++ base64 -d'
2026-03-14 09:44:27.404279 | controller |     - '    logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.404283 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:44:27.404302 | controller |     - '    logger.go:42: 09:37:33 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''cafile
2026-03-14 09:44:27.404309 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'''
2026-03-14 09:44:27.404313 | controller |     - '    logger.go:42: 09:37:34 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']'''
2026-03-14 09:44:27.404317 | controller |     - '    logger.go:42: 09:37:34 | watcher-tls/1-deploy-with-tlse | + grep infra-optim'
2026-03-14 09:44:27.404321 | controller |     - '    logger.go:42: 09:37:34 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.404325 | controller |       openstackclient -- openstack endpoint list'
2026-03-14 09:44:27.404330 | controller |     - '    logger.go:42: 09:37:34 | watcher-tls/1-deploy-with-tlse | ++ grep -c https'
2026-03-14 09:44:27.404334 | controller |     - '    logger.go:42: 09:37:35 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']'''
2026-03-14 09:44:27.404338 | controller |     - '    logger.go:42: 09:37:35 | watcher-tls/1-deploy-with-tlse | ++ oc get pods -n
2026-03-14 09:44:27.404342 | controller |       openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:44:27.404346 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/1-deploy-with-tlse | + ''['' '''' == ''''
2026-03-14 09:44:27.404351 | controller |       '']'''
2026-03-14 09:44:27.404355 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/1-deploy-with-tlse | + exit 0'
2026-03-14 09:44:27.404359 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/1-deploy-with-tlse | test step completed
2026-03-14 09:44:27.404363 | controller |       1-deploy-with-tlse'
2026-03-14 09:44:27.404367 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | starting test step 2-patch-mtls'
2026-03-14 09:44:27.404372 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | running command: [sh -c
2026-03-14 09:44:27.404376 | controller |       set -euxo pipefail'
2026-03-14 09:44:27.404380 | controller |     - '        oc patch oscp -n $NAMESPACE openstack --type=''json'' -p=''[{"op": "replace",
2026-03-14 09:44:27.404384 | controller |       "path": "/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode", "value": "Request"}]'''
2026-03-14 09:44:27.404388 | controller |     - '        ]'
2026-03-14 09:44:27.404392 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | + oc patch oscp -n watcher-kuttl-default
2026-03-14 09:44:27.404397 | controller |       openstack --type=json ''-p=[{"op": "replace", "path": "/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode",
2026-03-14 09:44:27.404401 | controller |       "value": "Request"}]'''
2026-03-14 09:44:27.404405 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | openstackcontrolplane.core.openstack.org/openstack
2026-03-14 09:44:27.404409 | controller |       patched'
2026-03-14 09:44:27.404413 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | running command: [sh -c
2026-03-14 09:44:27.404418 | controller |       set -euxo pipefail'
2026-03-14 09:44:27.404422 | controller |     - '        '
2026-03-14 09:44:27.404426 | controller |     - '        oc project ${NAMESPACE}'
2026-03-14 09:44:27.404430 | controller |     - '        # Get pod names for each watcher service'
2026-03-14 09:44:27.404434 | controller |     - '        APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:44:27.404439 | controller |     - '        APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:44:27.404443 | controller |     - '        DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:44:27.404447 | controller |     - '        '
2026-03-14 09:44:27.404451 | controller |     - '        # Verify memcached mTLS config parameters in watcher-api config'
2026-03-14 09:44:27.404461 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.404466 | controller |     - '          echo "Checking watcher-api config..."'
2026-03-14 09:44:27.404470 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404474 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:44:27.404478 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404484 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.404489 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404493 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.404497 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404501 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:44:27.404505 | controller |     - '        '
2026-03-14 09:44:27.404509 | controller |     - '          # Verify mTLS config parameters in memcached backend config'
2026-03-14 09:44:27.404514 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404518 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:44:27.404522 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404526 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.404530 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404535 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.404539 | controller |     - '        else'
2026-03-14 09:44:27.404543 | controller |     - '          exit 1'
2026-03-14 09:44:27.404547 | controller |     - '        fi'
2026-03-14 09:44:27.404554 | controller |     - '        '
2026-03-14 09:44:27.404558 | controller |     - '        # Verify memcached mTLS config parameters in watcher-applier config'
2026-03-14 09:44:27.404563 | controller |     - '        if [ -n "${APPLIERPOD}" ]; then'
2026-03-14 09:44:27.404567 | controller |     - '          echo "Checking watcher-applier config..."'
2026-03-14 09:44:27.404571 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404575 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:44:27.404579 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404584 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.404588 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404592 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.404596 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404600 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:44:27.404604 | controller |     - '        '
2026-03-14 09:44:27.404609 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404613 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:44:27.404617 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404621 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.404625 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404630 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.404634 | controller |     - '        else'
2026-03-14 09:44:27.404638 | controller |     - '          exit 1'
2026-03-14 09:44:27.404642 | controller |     - '        fi'
2026-03-14 09:44:27.404646 | controller |     - '        '
2026-03-14 09:44:27.404650 | controller |     - '        # Verify memcached mTLS config parameters in watcher-decision-engine config'
2026-03-14 09:44:27.404655 | controller |     - '        if [ -n "${DECISIONENGINEPOD}" ]; then'
2026-03-14 09:44:27.404660 | controller |     - '          echo "Checking watcher-decision-engine config..."'
2026-03-14 09:44:27.404665 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404669 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:44:27.404673 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404679 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.404683 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404687 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.404691 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404701 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:44:27.404705 | controller |     - '        '
2026-03-14 09:44:27.404710 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404714 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:44:27.404718 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404722 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.404726 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.404730 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.404735 | controller |     - '        else'
2026-03-14 09:44:27.404739 | controller |     - '          exit 1'
2026-03-14 09:44:27.404743 | controller |     - '        fi'
2026-03-14 09:44:27.404747 | controller |     - '        ]'
2026-03-14 09:44:27.404751 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default'
2026-03-14 09:44:27.404755 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default"
2026-03-14 09:44:27.404760 | controller |       on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.404764 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api
2026-03-14 09:44:27.404768 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:44:27.404772 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.404776 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier
2026-03-14 09:44:27.404780 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:44:27.404785 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0'
2026-03-14 09:44:27.404789 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine
2026-03-14 09:44:27.404793 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:44:27.404797 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.404801 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0
2026-03-14 09:44:27.404805 | controller |       '']'''
2026-03-14 09:44:27.404810 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api
2026-03-14 09:44:27.404814 | controller |       config...'''
2026-03-14 09:44:27.404818 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | Checking watcher-api config...'
2026-03-14 09:44:27.404822 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile
2026-03-14 09:44:27.404830 | controller |       = /etc/pki/tls/certs/mtls.crt '''
2026-03-14 09:44:27.404835 | controller |     - '    logger.go:42: 09:37:36 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.404839 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.404843 | controller |     - '    logger.go:42: 09:37:37 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.404847 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.404851 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.404856 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.404860 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.404866 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.404870 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.404874 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.404878 | controller |       project_domain_name = Default project_name = service user_domain_name = Default
2026-03-14 09:44:27.404882 | controller |       password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000
2026-03-14 09:44:27.404886 | controller |       interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.404891 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.404895 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.404899 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.404903 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.404913 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.404918 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.404924 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.404931 | controller |       tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc
2026-03-14 09:44:27.404938 | controller |       port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]''
2026-03-14 09:44:27.404945 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.404952 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.404958 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.404963 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.404969 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.404974 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.404980 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.404985 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.404994 | controller |     - '    logger.go:42: 09:37:37 | watcher-tls/2-patch-mtls | + ''['' 0 == 1 '']'''
2026-03-14 09:44:27.404999 | controller |     - '    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | running command: [sh -c
2026-03-14 09:44:27.405005 | controller |       set -euxo pipefail'
2026-03-14 09:44:27.405011 | controller |     - '        '
2026-03-14 09:44:27.405017 | controller |     - '        oc project ${NAMESPACE}'
2026-03-14 09:44:27.405022 | controller |     - '        # Get pod names for each watcher service'
2026-03-14 09:44:27.405026 | controller |     - '        APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:44:27.405033 | controller |     - '        APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:44:27.405037 | controller |     - '        DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:44:27.405041 | controller |     - '        '
2026-03-14 09:44:27.405046 | controller |     - '        # Verify memcached mTLS config parameters in watcher-api config'
2026-03-14 09:44:27.405050 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.405054 | controller |     - '          echo "Checking watcher-api config..."'
2026-03-14 09:44:27.405058 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405062 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:44:27.405067 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405071 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.405075 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405079 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.405083 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405088 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:44:27.405092 | controller |     - '        '
2026-03-14 09:44:27.405096 | controller |     - '          # Verify mTLS config parameters in memcached backend config'
2026-03-14 09:44:27.405100 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405104 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:44:27.405108 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405112 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.405117 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405121 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.405125 | controller |     - '        else'
2026-03-14 09:44:27.405129 | controller |     - '          exit 1'
2026-03-14 09:44:27.405133 | controller |     - '        fi'
2026-03-14 09:44:27.405138 | controller |     - '        '
2026-03-14 09:44:27.405142 | controller |     - '        # Verify memcached mTLS config parameters in watcher-applier config'
2026-03-14 09:44:27.405146 | controller |     - '        if [ -n "${APPLIERPOD}" ]; then'
2026-03-14 09:44:27.405150 | controller |     - '          echo "Checking watcher-applier config..."'
2026-03-14 09:44:27.405154 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405159 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:44:27.405163 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405174 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.405179 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405185 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.405190 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405194 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:44:27.405198 | controller |     - '        '
2026-03-14 09:44:27.405202 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405206 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:44:27.405211 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405215 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.405221 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405225 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.405229 | controller |     - '        else'
2026-03-14 09:44:27.405233 | controller |     - '          exit 1'
2026-03-14 09:44:27.405238 | controller |     - '        fi'
2026-03-14 09:44:27.405242 | controller |     - '        '
2026-03-14 09:44:27.405246 | controller |     - '        # Verify memcached mTLS config parameters in watcher-decision-engine config'
2026-03-14 09:44:27.405250 | controller |     - '        if [ -n "${DECISIONENGINEPOD}" ]; then'
2026-03-14 09:44:27.405254 | controller |     - '          echo "Checking watcher-decision-engine config..."'
2026-03-14 09:44:27.405260 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405264 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:44:27.405268 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405273 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.405277 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405283 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.405301 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405308 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:44:27.405312 | controller |     - '        '
2026-03-14 09:44:27.405317 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405321 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:44:27.405325 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405329 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.405333 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405338 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.405342 | controller |     - '        else'
2026-03-14 09:44:27.405346 | controller |     - '          exit 1'
2026-03-14 09:44:27.405350 | controller |     - '        fi'
2026-03-14 09:44:27.405354 | controller |     - '        ]'
2026-03-14 09:44:27.405359 | controller |     - '    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default'
2026-03-14 09:44:27.405363 | controller |     - '    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default"
2026-03-14 09:44:27.405367 | controller |       on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.405373 | controller |     - '    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api
2026-03-14 09:44:27.405380 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:44:27.405384 | controller |     - '    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.405388 | controller |     - '    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier
2026-03-14 09:44:27.405392 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:44:27.405397 | controller |     - '    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0'
2026-03-14 09:44:27.405401 | controller |     - '    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine
2026-03-14 09:44:27.405405 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:44:27.405409 | controller |     - '    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.405413 | controller |     - '    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0
2026-03-14 09:44:27.405418 | controller |       '']'''
2026-03-14 09:44:27.405424 | controller |     - '    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api
2026-03-14 09:44:27.405428 | controller |       config...'''
2026-03-14 09:44:27.405433 | controller |     - '    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | Checking watcher-api config...'
2026-03-14 09:44:27.405443 | controller |     - '    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile
2026-03-14 09:44:27.405448 | controller |       = /etc/pki/tls/certs/mtls.crt '''
2026-03-14 09:44:27.405452 | controller |     - '    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.405456 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.405461 | controller |     - '    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | error: cannot exec into
2026-03-14 09:44:27.405465 | controller |       a container in a completed pod; current phase is Failed'
2026-03-14 09:44:27.405469 | controller |     - '    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | ++ echo'
2026-03-14 09:44:27.405473 | controller |     - '    logger.go:42: 09:37:38 | watcher-tls/2-patch-mtls | + ''['' 0 == 1 '']'''
2026-03-14 09:44:27.405477 | controller |     - '    logger.go:42: 09:37:39 | watcher-tls/2-patch-mtls | running command: [sh -c
2026-03-14 09:44:27.405483 | controller |       set -euxo pipefail'
2026-03-14 09:44:27.405488 | controller |     - '        '
2026-03-14 09:44:27.405492 | controller |     - '        oc project ${NAMESPACE}'
2026-03-14 09:44:27.405496 | controller |     - '        # Get pod names for each watcher service'
2026-03-14 09:44:27.405500 | controller |     - '        APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:44:27.405504 | controller |     - '        APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:44:27.405509 | controller |     - '        DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:44:27.405514 | controller |     - '        '
2026-03-14 09:44:27.405518 | controller |     - '        # Verify memcached mTLS config parameters in watcher-api config'
2026-03-14 09:44:27.405522 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.405526 | controller |     - '          echo "Checking watcher-api config..."'
2026-03-14 09:44:27.405530 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405535 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:44:27.405539 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405543 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.405547 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405553 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.405557 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405561 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:44:27.405565 | controller |     - '        '
2026-03-14 09:44:27.405570 | controller |     - '          # Verify mTLS config parameters in memcached backend config'
2026-03-14 09:44:27.405576 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405580 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:44:27.405585 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405589 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.405593 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405597 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.405601 | controller |     - '        else'
2026-03-14 09:44:27.405605 | controller |     - '          exit 1'
2026-03-14 09:44:27.405610 | controller |     - '        fi'
2026-03-14 09:44:27.405614 | controller |     - '        '
2026-03-14 09:44:27.405618 | controller |     - '        # Verify memcached mTLS config parameters in watcher-applier config'
2026-03-14 09:44:27.405622 | controller |     - '        if [ -n "${APPLIERPOD}" ]; then'
2026-03-14 09:44:27.405626 | controller |     - '          echo "Checking watcher-applier config..."'
2026-03-14 09:44:27.405631 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405635 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:44:27.405639 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405643 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.405647 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405652 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.405656 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405660 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:44:27.405664 | controller |     - '        '
2026-03-14 09:44:27.405668 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405673 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:44:27.405677 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405686 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.405691 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405695 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.405699 | controller |     - '        else'
2026-03-14 09:44:27.405703 | controller |     - '          exit 1'
2026-03-14 09:44:27.405708 | controller |     - '        fi'
2026-03-14 09:44:27.405712 | controller |     - '        '
2026-03-14 09:44:27.405716 | controller |     - '        # Verify memcached mTLS config parameters in watcher-decision-engine config'
2026-03-14 09:44:27.405720 | controller |     - '        if [ -n "${DECISIONENGINEPOD}" ]; then'
2026-03-14 09:44:27.405724 | controller |     - '          echo "Checking watcher-decision-engine config..."'
2026-03-14 09:44:27.405729 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405733 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:44:27.405739 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405743 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.405747 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405751 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.405756 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405760 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:44:27.405764 | controller |     - '        '
2026-03-14 09:44:27.405768 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405772 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:44:27.405776 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405781 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.405785 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405789 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.405793 | controller |     - '        else'
2026-03-14 09:44:27.405797 | controller |     - '          exit 1'
2026-03-14 09:44:27.405802 | controller |     - '        fi'
2026-03-14 09:44:27.405806 | controller |     - '        ]'
2026-03-14 09:44:27.405810 | controller |     - '    logger.go:42: 09:37:39 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default'
2026-03-14 09:44:27.405814 | controller |     - '    logger.go:42: 09:37:39 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default"
2026-03-14 09:44:27.405819 | controller |       on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.405823 | controller |     - '    logger.go:42: 09:37:39 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api
2026-03-14 09:44:27.405827 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:44:27.405831 | controller |     - '    logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.405835 | controller |     - '    logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier
2026-03-14 09:44:27.405839 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:44:27.405844 | controller |     - '    logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0'
2026-03-14 09:44:27.405848 | controller |     - '    logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine
2026-03-14 09:44:27.405852 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:44:27.405856 | controller |     - '    logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.405860 | controller |     - '    logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0
2026-03-14 09:44:27.405865 | controller |       '']'''
2026-03-14 09:44:27.405869 | controller |     - '    logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api
2026-03-14 09:44:27.405873 | controller |       config...'''
2026-03-14 09:44:27.405877 | controller |     - '    logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls | Checking watcher-api config...'
2026-03-14 09:44:27.405882 | controller |     - '    logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile
2026-03-14 09:44:27.405886 | controller |       = /etc/pki/tls/certs/mtls.crt '''
2026-03-14 09:44:27.405890 | controller |     - '    logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.405894 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.405900 | controller |     - '    logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls | error: unable to upgrade
2026-03-14 09:44:27.405905 | controller |       connection: container not found ("watcher-api")'
2026-03-14 09:44:27.405909 | controller |     - '    logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls | ++ echo'
2026-03-14 09:44:27.405913 | controller |     - '    logger.go:42: 09:37:40 | watcher-tls/2-patch-mtls | + ''['' 0 == 1 '']'''
2026-03-14 09:44:27.405918 | controller |     - '    logger.go:42: 09:37:41 | watcher-tls/2-patch-mtls | running command: [sh -c
2026-03-14 09:44:27.405923 | controller |       set -euxo pipefail'
2026-03-14 09:44:27.405933 | controller |     - '        '
2026-03-14 09:44:27.405938 | controller |     - '        oc project ${NAMESPACE}'
2026-03-14 09:44:27.405944 | controller |     - '        # Get pod names for each watcher service'
2026-03-14 09:44:27.405949 | controller |     - '        APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:44:27.405953 | controller |     - '        APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:44:27.405957 | controller |     - '        DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:44:27.405961 | controller |     - '        '
2026-03-14 09:44:27.405965 | controller |     - '        # Verify memcached mTLS config parameters in watcher-api config'
2026-03-14 09:44:27.405970 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.405974 | controller |     - '          echo "Checking watcher-api config..."'
2026-03-14 09:44:27.405978 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405982 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:44:27.405987 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.405994 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.405998 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.406002 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.406007 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.406011 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:44:27.406015 | controller |     - '        '
2026-03-14 09:44:27.406020 | controller |     - '          # Verify mTLS config parameters in memcached backend config'
2026-03-14 09:44:27.406024 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.406028 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:44:27.406033 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.406037 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.406041 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.406045 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.406049 | controller |     - '        else'
2026-03-14 09:44:27.406053 | controller |     - '          exit 1'
2026-03-14 09:44:27.406058 | controller |     - '        fi'
2026-03-14 09:44:27.406062 | controller |     - '        '
2026-03-14 09:44:27.406066 | controller |     - '        # Verify memcached mTLS config parameters in watcher-applier config'
2026-03-14 09:44:27.406070 | controller |     - '        if [ -n "${APPLIERPOD}" ]; then'
2026-03-14 09:44:27.406074 | controller |     - '          echo "Checking watcher-applier config..."'
2026-03-14 09:44:27.406079 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.406083 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:44:27.406089 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.406093 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.406097 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.406101 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.406105 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.406110 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:44:27.406114 | controller |     - '        '
2026-03-14 09:44:27.406118 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.406122 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:44:27.406126 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.406130 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.406138 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.406166 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.406170 | controller |     - '        else'
2026-03-14 09:44:27.406174 | controller |     - '          exit 1'
2026-03-14 09:44:27.406179 | controller |     - '        fi'
2026-03-14 09:44:27.406183 | controller |     - '        '
2026-03-14 09:44:27.406189 | controller |     - '        # Verify memcached mTLS config parameters in watcher-decision-engine config'
2026-03-14 09:44:27.406193 | controller |     - '        if [ -n "${DECISIONENGINEPOD}" ]; then'
2026-03-14 09:44:27.406198 | controller |     - '          echo "Checking watcher-decision-engine config..."'
2026-03-14 09:44:27.406202 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.406206 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:44:27.406217 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.406221 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.406226 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.406230 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.406234 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.406238 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:44:27.406242 | controller |     - '        '
2026-03-14 09:44:27.406247 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.406251 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:44:27.406255 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.406259 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.406263 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.406268 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.406272 | controller |     - '        else'
2026-03-14 09:44:27.406276 | controller |     - '          exit 1'
2026-03-14 09:44:27.406280 | controller |     - '        fi'
2026-03-14 09:44:27.406284 | controller |     - '        ]'
2026-03-14 09:44:27.406299 | controller |     - '    logger.go:42: 09:37:41 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default'
2026-03-14 09:44:27.406308 | controller |     - '    logger.go:42: 09:37:41 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default"
2026-03-14 09:44:27.406312 | controller |       on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.406316 | controller |     - '    logger.go:42: 09:37:41 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api
2026-03-14 09:44:27.406320 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:44:27.406324 | controller |     - '    logger.go:42: 09:37:41 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.406329 | controller |     - '    logger.go:42: 09:37:41 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier
2026-03-14 09:44:27.406334 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:44:27.406338 | controller |     - '    logger.go:42: 09:37:41 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0'
2026-03-14 09:44:27.406342 | controller |     - '    logger.go:42: 09:37:41 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine
2026-03-14 09:44:27.406346 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:44:27.406350 | controller |     - '    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.406355 | controller |     - '    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0
2026-03-14 09:44:27.406359 | controller |       '']'''
2026-03-14 09:44:27.406363 | controller |     - '    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api
2026-03-14 09:44:27.406367 | controller |       config...'''
2026-03-14 09:44:27.406371 | controller |     - '    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | Checking watcher-api config...'
2026-03-14 09:44:27.406376 | controller |     - '    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.406380 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.406384 | controller |     - '    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile
2026-03-14 09:44:27.406388 | controller |       = /etc/pki/tls/certs/mtls.crt '''
2026-03-14 09:44:27.406392 | controller |     - '    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.406397 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.406401 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.406405 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.406409 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.406413 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.406420 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.406424 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.406428 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.406432 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.406442 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.406447 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.406451 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.406455 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.406461 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.406466 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.406470 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.406474 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.406478 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.406482 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.406487 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.406491 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.406495 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.406499 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.406503 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.406508 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.406512 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.406516 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.406520 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.406524 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.406528 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.406532 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.406538 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.406543 | controller |     - '    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.406547 | controller |     - '    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile
2026-03-14 09:44:27.406551 | controller |       = /etc/pki/tls/private/mtls.key'''
2026-03-14 09:44:27.406555 | controller |     - '    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.406560 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.406564 | controller |     - '    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.406569 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.406574 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.406578 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.406582 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.406586 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.406592 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.406596 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.406605 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.406610 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.406614 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.406618 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.406622 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.406626 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.406635 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.406640 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.406644 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.406648 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.406652 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.406657 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.406661 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.406668 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.406672 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.406676 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.406680 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.406684 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.406689 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.406693 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.406697 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.406701 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.406705 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.406710 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.406714 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.406718 | controller |     - '    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.406722 | controller |     - '    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile
2026-03-14 09:44:27.406726 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'''
2026-03-14 09:44:27.406731 | controller |     - '    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.406735 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.406739 | controller |     - '    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.406743 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.406747 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.406753 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.406757 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.406762 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.406766 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.406770 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.406774 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.406779 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.406783 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.406787 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.406791 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.406795 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.406799 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.406803 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.406808 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.406812 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.406821 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.406826 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.406830 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.406836 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.406841 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.406847 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.406852 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.406857 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.406869 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.406881 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.406888 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.406894 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.406900 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.406906 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.406911 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.406916 | controller |     - '    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.406923 | controller |     - '    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled
2026-03-14 09:44:27.406935 | controller |       = true'''
2026-03-14 09:44:27.406942 | controller |     - '    logger.go:42: 09:37:42 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.406949 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.406955 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.406961 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.406966 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.406973 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.406979 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.406985 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.406991 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.406997 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.407003 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.407008 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.407013 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.407018 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.407022 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.407026 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.407030 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.407034 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.407038 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.407043 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.407047 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.407051 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.407055 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.407059 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.407072 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.407077 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.407081 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.407085 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.407089 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.407093 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.407098 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.407102 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.407112 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.407118 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.407123 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.407129 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.407135 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt'
2026-03-14 09:44:27.407141 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.407150 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.407157 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.407163 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.407168 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.407174 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.407180 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.407186 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.407192 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.407198 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.407204 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.407210 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.407215 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.407224 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.407230 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.407236 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.407242 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.407247 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.407253 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.407259 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.407265 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.407271 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.407277 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.407282 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.407308 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.407315 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.407324 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.407330 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.407344 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.407351 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.407356 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.407362 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.407368 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.407373 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.407379 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.407385 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.407391 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key'
2026-03-14 09:44:27.407397 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.407403 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.407408 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.407414 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.407420 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.407429 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.407436 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.407441 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.407447 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.407453 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.407459 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.407465 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.407470 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.407476 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.407482 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.407488 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.407494 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.407500 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.407506 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.407512 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.407517 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.407526 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.407532 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.407537 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.407543 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.407548 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.407552 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.407556 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.407560 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.407564 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.407571 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.407575 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.407579 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.407591 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.407596 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.407600 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.407604 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt'
2026-03-14 09:44:27.407609 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.407613 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.407617 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.407621 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.407625 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.407629 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.407634 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.407638 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.407642 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.407646 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.407650 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.407655 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.407659 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.407663 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.407667 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.407673 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.407677 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.407681 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.407686 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.407690 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.407694 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.407698 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.407702 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.407706 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.407711 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.407715 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.407719 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.407723 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.407727 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.407731 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.407736 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.407740 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.407744 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.407748 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.407752 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.407756 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.407761 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-applier-0
2026-03-14 09:44:27.407765 | controller |       '']'''
2026-03-14 09:44:27.407774 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-applier
2026-03-14 09:44:27.407779 | controller |       config...'''
2026-03-14 09:44:27.407783 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | Checking watcher-applier
2026-03-14 09:44:27.407788 | controller |       config...'
2026-03-14 09:44:27.407792 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile
2026-03-14 09:44:27.407796 | controller |       = /etc/pki/tls/certs/mtls.crt '''
2026-03-14 09:44:27.407800 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.407804 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.407808 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.407813 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.407817 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.407821 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.407827 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.407832 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.407836 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.407840 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.407867 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.407871 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.407876 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.407880 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.407884 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.407888 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.407895 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.407899 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.407905 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.407909 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.407913 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.407917 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.407921 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.407926 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:44:27.407930 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.407934 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.407938 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.407943 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.407947 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.407951 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.407955 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.407960 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.407964 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.407968 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile
2026-03-14 09:44:27.407972 | controller |       = /etc/pki/tls/private/mtls.key'''
2026-03-14 09:44:27.407977 | controller |     - '    logger.go:42: 09:37:43 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.407981 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.407991 | controller |     - '    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.407995 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.408002 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.408006 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.408011 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.408017 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.408023 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.408029 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.408035 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.408041 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.408047 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.408056 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.408061 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.408066 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.408070 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.408074 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.408078 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.408083 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.408087 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.408091 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.408095 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.408100 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:44:27.408104 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.408109 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.408115 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.408122 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.408128 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.408134 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.408140 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.408146 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.408152 | controller |     - '    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.408157 | controller |     - '    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile
2026-03-14 09:44:27.408161 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'''
2026-03-14 09:44:27.408168 | controller |     - '    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.408172 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.408176 | controller |     - '    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.408181 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.408185 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.408189 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.408203 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.408208 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.408214 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.408227 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.408238 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.408244 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.408250 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.408255 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.408261 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.408267 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.408273 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.408278 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.408282 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.408308 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.408313 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.408318 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.408322 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.408327 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:44:27.408331 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.408335 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.408339 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.408343 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.408347 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.408352 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.408356 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.408365 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.408369 | controller |     - '    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.408373 | controller |     - '    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.408377 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.408381 | controller |     - '    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled
2026-03-14 09:44:27.408385 | controller |       = true'''
2026-03-14 09:44:27.408390 | controller |     - '    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.408394 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.408398 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.408402 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.408406 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.408411 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.408415 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.408422 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.408426 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.408430 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.408434 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.408447 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.408453 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.408470 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.408477 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.408483 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.408488 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.408494 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.408500 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.408506 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.408512 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.408517 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:44:27.408521 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.408525 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.408529 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.408534 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.408544 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.408549 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.408553 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.408557 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.408562 | controller |     - '    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.408566 | controller |     - '    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt'
2026-03-14 09:44:27.408570 | controller |     - '    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.408575 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.408579 | controller |     - '    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.408584 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.408588 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.408594 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.408598 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.408603 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.408607 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.408611 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.408615 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.408619 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.408624 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.408628 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.408632 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.408636 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.408640 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.408645 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.408649 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.408662 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.408667 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.408672 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.408676 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.408680 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:44:27.408684 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.408690 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.408695 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.408699 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.408703 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.408707 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.408711 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.408716 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.408720 | controller |     - '    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.408724 | controller |     - '    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key'
2026-03-14 09:44:27.408728 | controller |     - '    logger.go:42: 09:37:44 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.408733 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.408737 | controller |     - '    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.408741 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.408745 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.408750 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.408754 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.408758 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.408763 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.408767 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.408771 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.408775 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.408780 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.408784 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.408788 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.408792 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.408796 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.408801 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.408805 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.408809 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.408813 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.408819 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.408826 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.408834 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:44:27.408840 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.408847 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.408859 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.408866 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.408873 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.408877 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.408881 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.408886 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.408890 | controller |     - '    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.408894 | controller |     - '    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt'
2026-03-14 09:44:27.408898 | controller |     - '    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.408903 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.408907 | controller |     - '    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.408911 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.408915 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.408920 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.408925 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.408931 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.408937 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.408943 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.408950 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.408956 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.408962 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.408968 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.408979 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.408984 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.408988 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.408993 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.408999 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.409003 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.409009 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.409014 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.409019 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.409025 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:44:27.409031 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.409037 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.409044 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.409050 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.409056 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.409062 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.409067 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.409073 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.409086 | controller |     - '    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.409092 | controller |     - '    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.409096 | controller |       '']'''
2026-03-14 09:44:27.409103 | controller |     - '    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-decision-engine
2026-03-14 09:44:27.409108 | controller |       config...'''
2026-03-14 09:44:27.409113 | controller |     - '    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | Checking watcher-decision-engine
2026-03-14 09:44:27.409119 | controller |       config...'
2026-03-14 09:44:27.409125 | controller |     - '    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile
2026-03-14 09:44:27.409132 | controller |       = /etc/pki/tls/certs/mtls.crt '''
2026-03-14 09:44:27.409138 | controller |     - '    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:44:27.409144 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.409150 | controller |     - '    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | error: Internal error occurred:
2026-03-14 09:44:27.409156 | controller |       error executing command in container: container is not created or running'
2026-03-14 09:44:27.409162 | controller |     - '    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | ++ echo'
2026-03-14 09:44:27.409168 | controller |     - '    logger.go:42: 09:37:45 | watcher-tls/2-patch-mtls | + ''['' 0 == 1 '']'''
2026-03-14 09:44:27.409174 | controller |     - '    logger.go:42: 09:37:46 | watcher-tls/2-patch-mtls | running command: [sh -c
2026-03-14 09:44:27.409180 | controller |       set -euxo pipefail'
2026-03-14 09:44:27.409185 | controller |     - '        '
2026-03-14 09:44:27.409191 | controller |     - '        oc project ${NAMESPACE}'
2026-03-14 09:44:27.409197 | controller |     - '        # Get pod names for each watcher service'
2026-03-14 09:44:27.409205 | controller |     - '        APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:44:27.409211 | controller |     - '        APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:44:27.409217 | controller |     - '        DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')'
2026-03-14 09:44:27.409223 | controller |     - '        '
2026-03-14 09:44:27.409230 | controller |     - '        # Verify memcached mTLS config parameters in watcher-api config'
2026-03-14 09:44:27.409239 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.409245 | controller |     - '          echo "Checking watcher-api config..."'
2026-03-14 09:44:27.409251 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409257 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:44:27.409263 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409269 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.409275 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409280 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.409303 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409313 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:44:27.409319 | controller |     - '        '
2026-03-14 09:44:27.409325 | controller |     - '          # Verify mTLS config parameters in memcached backend config'
2026-03-14 09:44:27.409331 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409337 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:44:27.409343 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409349 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.409355 | controller |     - '          [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409361 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.409366 | controller |     - '        else'
2026-03-14 09:44:27.409372 | controller |     - '          exit 1'
2026-03-14 09:44:27.409378 | controller |     - '        fi'
2026-03-14 09:44:27.409384 | controller |     - '        '
2026-03-14 09:44:27.409390 | controller |     - '        # Verify memcached mTLS config parameters in watcher-applier config'
2026-03-14 09:44:27.409396 | controller |     - '        if [ -n "${APPLIERPOD}" ]; then'
2026-03-14 09:44:27.409402 | controller |     - '          echo "Checking watcher-applier config..."'
2026-03-14 09:44:27.409408 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409414 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:44:27.409420 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409426 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.409432 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409437 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.409452 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409458 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:44:27.409464 | controller |     - '        '
2026-03-14 09:44:27.409470 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409475 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:44:27.409481 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409487 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.409496 | controller |     - '          [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409504 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.409508 | controller |     - '        else'
2026-03-14 09:44:27.409513 | controller |     - '          exit 1'
2026-03-14 09:44:27.409517 | controller |     - '        fi'
2026-03-14 09:44:27.409521 | controller |     - '        '
2026-03-14 09:44:27.409525 | controller |     - '        # Verify memcached mTLS config parameters in watcher-decision-engine config'
2026-03-14 09:44:27.409530 | controller |     - '        if [ -n "${DECISIONENGINEPOD}" ]; then'
2026-03-14 09:44:27.409534 | controller |     - '          echo "Checking watcher-decision-engine config..."'
2026-03-14 09:44:27.409538 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409542 | controller |       | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]'
2026-03-14 09:44:27.409546 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409551 | controller |       | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.409555 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409559 | controller |       | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.409563 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409567 | controller |       | grep -c "memcache_tls_enabled = true") == 1 ]'
2026-03-14 09:44:27.409571 | controller |     - '        '
2026-03-14 09:44:27.409576 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409580 | controller |       | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]'
2026-03-14 09:44:27.409584 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409588 | controller |       | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]'
2026-03-14 09:44:27.409592 | controller |     - '          [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.409596 | controller |       | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]'
2026-03-14 09:44:27.409601 | controller |     - '        else'
2026-03-14 09:44:27.409605 | controller |     - '          exit 1'
2026-03-14 09:44:27.409609 | controller |     - '        fi'
2026-03-14 09:44:27.409613 | controller |     - '        ]'
2026-03-14 09:44:27.409617 | controller |     - '    logger.go:42: 09:37:46 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default'
2026-03-14 09:44:27.409622 | controller |     - '    logger.go:42: 09:37:46 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default"
2026-03-14 09:44:27.409626 | controller |       on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.409630 | controller |     - '    logger.go:42: 09:37:46 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api
2026-03-14 09:44:27.409634 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:44:27.409638 | controller |     - '    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.409642 | controller |     - '    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier
2026-03-14 09:44:27.409647 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:44:27.409651 | controller |     - '    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0'
2026-03-14 09:44:27.409655 | controller |     - '    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine
2026-03-14 09:44:27.409659 | controller |       -o ''jsonpath={.items[0].metadata.name}'''
2026-03-14 09:44:27.409663 | controller |     - '    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.409668 | controller |     - '    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0
2026-03-14 09:44:27.409675 | controller |       '']'''
2026-03-14 09:44:27.409679 | controller |     - '    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api
2026-03-14 09:44:27.409683 | controller |       config...'''
2026-03-14 09:44:27.409687 | controller |     - '    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | Checking watcher-api config...'
2026-03-14 09:44:27.409692 | controller |     - '    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile
2026-03-14 09:44:27.409696 | controller |       = /etc/pki/tls/certs/mtls.crt '''
2026-03-14 09:44:27.409700 | controller |     - '    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.409710 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.409714 | controller |     - '    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.409719 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.409723 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.409727 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.409731 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.409737 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.409741 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.409745 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.409750 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.409754 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.409758 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.409762 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.409767 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.409771 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.409775 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.409779 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.409783 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.409787 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.409791 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.409795 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.409800 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.409804 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.409808 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.409812 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.409816 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.409822 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.409826 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.409830 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.409836 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.409840 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.409844 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.409849 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.409853 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.409857 | controller |     - '    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.409861 | controller |     - '    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile
2026-03-14 09:44:27.409865 | controller |       = /etc/pki/tls/private/mtls.key'''
2026-03-14 09:44:27.409869 | controller |     - '    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.409874 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.409878 | controller |     - '    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.409882 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.409886 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.409896 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.409901 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.409905 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.409909 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.409914 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.409918 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.409922 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.409926 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.409930 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.409934 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.409938 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.409943 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.409947 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.409951 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.409955 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.409959 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.409965 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.409969 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.409973 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.409978 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.409982 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.409986 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.409990 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.409994 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.409998 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.410002 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.410006 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.410011 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.410015 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.410019 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.410023 | controller |     - '    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.410027 | controller |     - '    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile
2026-03-14 09:44:27.410032 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'''
2026-03-14 09:44:27.410036 | controller |     - '    logger.go:42: 09:37:47 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.410040 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.410044 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.410048 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.410053 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.410057 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.410061 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.410065 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.410074 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.410079 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.410083 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.410087 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.410091 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.410095 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.410099 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.410105 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.410109 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.410113 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.410118 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.410123 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.410129 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.410133 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.410138 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.410142 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.410159 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.410164 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.410168 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.410172 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.410176 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.410180 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.410185 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.410189 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.410193 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.410197 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.410201 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.410205 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.410210 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled
2026-03-14 09:44:27.410214 | controller |       = true'''
2026-03-14 09:44:27.410218 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.410222 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.410226 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.410231 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.410235 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.410240 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.410246 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.410251 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.410255 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.410259 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.410265 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.410274 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.410279 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.410283 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.410299 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.410306 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.410310 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.410314 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.410318 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.410323 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.410327 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.410331 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.410335 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.410339 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.410344 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.410348 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.410352 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.410356 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.410360 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.410364 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.410369 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.410373 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.410377 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.410381 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.410385 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.410390 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.410394 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt'
2026-03-14 09:44:27.410398 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.410402 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.410406 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.410411 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.410415 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.410421 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.410425 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.410429 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.410434 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.410438 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.410442 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.410464 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.410468 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.410473 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.410477 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.410481 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.410491 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.410496 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.410500 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.410504 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.410508 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.410512 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.410517 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.410521 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.410525 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.410529 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.410533 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.410538 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.410542 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.410546 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.410550 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.410554 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.410558 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.410563 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.410567 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.410571 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.410575 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key'
2026-03-14 09:44:27.410581 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.410586 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.410590 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.410594 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.410600 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.410604 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.410608 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.410613 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.410617 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.410621 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.410625 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.410629 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.410634 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.410638 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.410642 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.410646 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.410650 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.410655 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.410659 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.410663 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.410672 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.410676 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.410681 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.410687 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.410691 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.410695 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.410699 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.410703 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.410710 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.410715 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.410719 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.410723 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.410729 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.410733 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.410739 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.410743 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.410747 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt'
2026-03-14 09:44:27.410751 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api
2026-03-14 09:44:27.410755 | controller |       watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.410760 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.410764 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.410768 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.410772 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.410777 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.410781 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.410785 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.410789 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.410793 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.410797 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.410802 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.410806 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.410810 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.410814 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.410818 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.410822 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.410827 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.410831 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.410835 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.410839 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.410843 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.410847 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.410858 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.410863 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.410867 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.410873 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.410877 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.410881 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.410885 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.410889 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.410894 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.410898 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.410902 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.410906 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.410910 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-applier-0
2026-03-14 09:44:27.410915 | controller |       '']'''
2026-03-14 09:44:27.410919 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-applier
2026-03-14 09:44:27.410923 | controller |       config...'''
2026-03-14 09:44:27.410927 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | Checking watcher-applier
2026-03-14 09:44:27.410931 | controller |       config...'
2026-03-14 09:44:27.410935 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile
2026-03-14 09:44:27.410940 | controller |       = /etc/pki/tls/certs/mtls.crt '''
2026-03-14 09:44:27.410944 | controller |     - '    logger.go:42: 09:37:48 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.410948 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.410952 | controller |     - '    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.410957 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.410961 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.410966 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.410971 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.410976 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.410980 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.410985 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.410990 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.410995 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.410999 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.411005 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.411011 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.411015 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.411021 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.411025 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.411029 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.411033 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.411038 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.411042 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.411046 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.411050 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:44:27.411059 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.411064 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.411068 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.411072 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.411076 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.411080 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.411085 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.411089 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.411093 | controller |     - '    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.411097 | controller |     - '    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile
2026-03-14 09:44:27.411101 | controller |       = /etc/pki/tls/private/mtls.key'''
2026-03-14 09:44:27.411106 | controller |     - '    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.411110 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.411114 | controller |     - '    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.411118 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.411123 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.411127 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.411131 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.411135 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.411139 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.411143 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.411148 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.411153 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.411158 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.411177 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.411186 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.411191 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.411197 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.411202 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.411208 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.411214 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.411219 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.411225 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.411230 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.411236 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:44:27.411241 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.411247 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.411253 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.411258 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.411264 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.411269 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.411287 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.411315 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.411322 | controller |     - '    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.411328 | controller |     - '    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile
2026-03-14 09:44:27.411334 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'''
2026-03-14 09:44:27.411342 | controller |     - '    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.411347 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.411351 | controller |     - '    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.411355 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.411360 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.411364 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.411368 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.411372 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.411376 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.411384 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.411388 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.411392 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.411396 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.411400 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.411404 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.411409 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.411413 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.411417 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.411421 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.411425 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.411430 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.411434 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.411438 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.411442 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:44:27.411446 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.411450 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.411455 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.411459 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.411463 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.411467 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.411471 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.411475 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.411480 | controller |     - '    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.411484 | controller |     - '    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled
2026-03-14 09:44:27.411488 | controller |       = true'''
2026-03-14 09:44:27.411492 | controller |     - '    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.411496 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.411501 | controller |     - '    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.411512 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.411517 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.411521 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.411525 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.411538 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.411543 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.411547 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.411551 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.411556 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.411560 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.411564 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.411568 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.411573 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.411577 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.411581 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.411585 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.411590 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.411594 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.411598 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.411602 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.411607 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:44:27.411611 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.411617 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.411621 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.411625 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.411629 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.411634 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.411638 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.411642 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.411647 | controller |     - '    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.411651 | controller |     - '    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt'
2026-03-14 09:44:27.411655 | controller |     - '    logger.go:42: 09:37:49 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.411659 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.411663 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.411668 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.411672 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.411678 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.411699 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.411713 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.411718 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.411723 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.411727 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.411731 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.411736 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.411740 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.411744 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.411749 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.411753 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.411757 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.411761 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.411766 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.411770 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.411774 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.411778 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.411783 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:44:27.411787 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.411791 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.411795 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.411799 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.411804 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.411808 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.411812 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.411816 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.411821 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.411825 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key'
2026-03-14 09:44:27.411829 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.411834 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.411840 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.411844 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.411848 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.411853 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.411857 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.411861 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.411866 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.411870 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.411874 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.411878 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.411882 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.411892 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.411897 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.411901 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.411905 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.411910 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.411914 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.411918 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.411922 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.411927 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.411931 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.411935 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:44:27.411939 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.411944 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.411948 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.411952 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.411956 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.411961 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.411965 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.411969 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.411973 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.411979 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier
2026-03-14 09:44:27.411984 | controller |       watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.411988 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt'
2026-03-14 09:44:27.411992 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.411996 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.412001 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.412005 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.412009 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.412014 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.412018 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.412025 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.412029 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.412033 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.412038 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.412042 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.412046 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.412050 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.412055 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.412059 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.412063 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.412067 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.412079 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.412084 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.412088 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.412092 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]''
2026-03-14 09:44:27.412096 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.412101 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.412105 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.412109 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.412113 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.412118 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.412122 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.412128 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.412132 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.412136 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-decision-engine-0
2026-03-14 09:44:27.412140 | controller |       '']'''
2026-03-14 09:44:27.412145 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-decision-engine
2026-03-14 09:44:27.412149 | controller |       config...'''
2026-03-14 09:44:27.412153 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | Checking watcher-decision-engine
2026-03-14 09:44:27.412158 | controller |       config...'
2026-03-14 09:44:27.412162 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile
2026-03-14 09:44:27.412166 | controller |       = /etc/pki/tls/certs/mtls.crt '''
2026-03-14 09:44:27.412171 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:44:27.412175 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.412179 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.412183 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.412188 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.412192 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.412196 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.412201 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.412205 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.412209 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.412213 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.412218 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.412222 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.412226 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.412230 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.412235 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.412239 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.412243 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.412247 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.412252 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.412256 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.412260 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.412270 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.412277 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.412281 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.412301 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.412308 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.412313 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.412317 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.412322 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.412326 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.412330 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.412334 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.412339 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.412343 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.412347 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile
2026-03-14 09:44:27.412351 | controller |       = /etc/pki/tls/private/mtls.key'''
2026-03-14 09:44:27.412358 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:44:27.412362 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.412366 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.412371 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.412375 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.412379 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.412383 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.412388 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.412392 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.412396 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.412401 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.412405 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.412409 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.412413 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.412417 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.412422 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.412426 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.412432 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.412436 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.412440 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.412444 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.412449 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.412453 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.412457 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.412463 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.412467 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.412472 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.412483 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.412489 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.412493 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.412498 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.412502 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.412507 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.412511 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.412515 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.412519 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile
2026-03-14 09:44:27.412524 | controller |       = /etc/pki/tls/certs/mtls-ca.crt'''
2026-03-14 09:44:27.412528 | controller |     - '    logger.go:42: 09:37:50 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:44:27.412532 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.412536 | controller |     - '    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.412541 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.412545 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.412549 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.412554 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.412558 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.412562 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.412567 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.412571 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.412575 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.412581 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.412585 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.412589 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.412594 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.412598 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.412602 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.412607 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.412611 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.412615 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.412619 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.412624 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.412628 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.412632 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.412637 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.412641 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.412645 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.412651 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.412662 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.412676 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.412696 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.412704 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.412711 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.412716 | controller |     - '    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.412720 | controller |     - '    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled
2026-03-14 09:44:27.412724 | controller |       = true'''
2026-03-14 09:44:27.412729 | controller |     - '    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:44:27.412733 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.412737 | controller |     - '    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.412741 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.412746 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.412750 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.412756 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.412765 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.412772 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.412778 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.412784 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.412790 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.412796 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.412802 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.412808 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.412813 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.412819 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.412823 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.412827 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.412831 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.412836 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.412840 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.412845 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.412850 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.412856 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.412863 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.412869 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.412875 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.412881 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.412887 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.412893 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.412899 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.412905 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.412910 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.412916 | controller |     - '    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.412930 | controller |     - '    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt'
2026-03-14 09:44:27.412935 | controller |     - '    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:44:27.412943 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.412949 | controller |     - '    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.412956 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.412966 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.412973 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.412979 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.412986 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.412994 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.412999 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.413005 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.413011 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.413017 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.413022 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.413028 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.413034 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.413040 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.413046 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.413052 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.413058 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.413064 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.413070 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.413075 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.413081 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.413087 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.413093 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.413098 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.413104 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.413110 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.413118 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.413124 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.413130 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.413136 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.413142 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.413148 | controller |     - '    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.413156 | controller |     - '    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key'
2026-03-14 09:44:27.413162 | controller |     - '    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:44:27.413168 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.413174 | controller |     - '    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.413187 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.413193 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.413199 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.413205 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.413211 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.413217 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.413223 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.413229 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.413235 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.413240 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.413246 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.413252 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.413258 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.413263 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.413269 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.413275 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.413281 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.413304 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.413314 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.413320 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.413330 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.413335 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.413341 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.413347 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.413353 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.413359 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.413364 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.413373 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.413379 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.413385 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.413391 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.413396 | controller |     - '    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.413402 | controller |     - '    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt'
2026-03-14 09:44:27.413408 | controller |     - '    logger.go:42: 09:37:51 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine
2026-03-14 09:44:27.413414 | controller |       watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.413420 | controller |     - '    logger.go:42: 09:37:52 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path
2026-03-14 09:44:27.413426 | controller |       = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.413431 | controller |       control_exchange = watcher debug = True ''#'' empty notification_level means that
2026-03-14 09:44:27.413437 | controller |       no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_6c35:2385b7d3506503ff893d33774928a31b@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.413451 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.413457 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.413463 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.413469 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.413475 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.413481 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.413487 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.413493 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.413499 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.413504 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.413513 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.413519 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.413525 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.413531 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.413536 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.413542 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.413548 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.413554 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.413560 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.413568 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type
2026-03-14 09:44:27.413574 | controller |       = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal
2026-03-14 09:44:27.413580 | controller |       region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.413586 | controller |       = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne
2026-03-14 09:44:27.413592 | controller |       ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.413597 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.413603 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.413609 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.413615 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.413621 | controller |     - '    logger.go:42: 09:37:52 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.413627 | controller |     - '    logger.go:42: 09:37:52 | watcher-tls/2-patch-mtls | test step completed 2-patch-mtls'
2026-03-14 09:44:27.413632 | controller |     - '    logger.go:42: 09:37:52 | watcher-tls/3-disable-podlevel-tls | starting test
2026-03-14 09:44:27.413638 | controller |       step 3-disable-podlevel-tls'
2026-03-14 09:44:27.413647 | controller |     - '    logger.go:42: 09:37:52 | watcher-tls/3-disable-podlevel-tls | running command:
2026-03-14 09:44:27.413653 | controller |       [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op":
2026-03-14 09:44:27.413659 | controller |       "replace", "path": "/spec/apiServiceTemplate/tls/api", "value":{ "internal": {},
2026-03-14 09:44:27.413665 | controller |       "public": {} }}]'''
2026-03-14 09:44:27.413671 | controller |     - '        ]'
2026-03-14 09:44:27.413677 | controller |     - '    logger.go:42: 09:37:52 | watcher-tls/3-disable-podlevel-tls | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:44:27.413683 | controller |       patched'
2026-03-14 09:44:27.413688 | controller |     - '    logger.go:42: 09:37:52 | watcher-tls/3-disable-podlevel-tls | running command:
2026-03-14 09:44:27.413694 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.413700 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.413705 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.413711 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.413716 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.413725 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.413731 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.413739 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.413744 | controller |       ]'
2026-03-14 09:44:27.413750 | controller |     - '        # check that watcher internal endpoint does not use https'
2026-03-14 09:44:27.413764 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.413770 | controller |       infra-optim | grep internal  | [ $(grep -c https) == 0 ]'
2026-03-14 09:44:27.413776 | controller |     - '        # check that watcher public endpoint does use https'
2026-03-14 09:44:27.413782 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.413788 | controller |       infra-optim | grep public  | [ $(grep -c https) == 1 ]'
2026-03-14 09:44:27.413793 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.413799 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.413805 | controller |       == "" ]; then'
2026-03-14 09:44:27.413811 | controller |     - '            exit 0'
2026-03-14 09:44:27.413820 | controller |     - '        fi'
2026-03-14 09:44:27.413826 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.413831 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.413836 | controller |     - '        counter=0'
2026-03-14 09:44:27.413842 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.413847 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.413853 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.413862 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.413867 | controller |     - '            fi'
2026-03-14 09:44:27.413873 | controller |     - '        done'
2026-03-14 09:44:27.413879 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.413884 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.413890 | controller |     - '          exit 1'
2026-03-14 09:44:27.413895 | controller |     - '        else'
2026-03-14 09:44:27.413901 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.413907 | controller |     - '        fi'
2026-03-14 09:44:27.413913 | controller |     - '        ]'
2026-03-14 09:44:27.413918 | controller |     - '    logger.go:42: 09:37:52 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:44:27.413924 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.413929 | controller |       -c Type'
2026-03-14 09:44:27.413935 | controller |     - '    logger.go:42: 09:37:52 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.413941 | controller |     - '    logger.go:42: 09:37:54 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:44:27.413947 | controller |       1 '']'''
2026-03-14 09:44:27.413953 | controller |     - '    logger.go:42: 09:37:54 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print
2026-03-14 09:44:27.413958 | controller |       $1}'''
2026-03-14 09:44:27.413964 | controller |     - '    logger.go:42: 09:37:54 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher'
2026-03-14 09:44:27.413970 | controller |     - '    logger.go:42: 09:37:54 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n
2026-03-14 09:44:27.413976 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.413982 | controller |       -c Type -c ID'
2026-03-14 09:44:27.413988 | controller |     - '    logger.go:42: 09:37:56 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=e0a908c724554093bb8de86515131a2c'
2026-03-14 09:44:27.413994 | controller |     - '    logger.go:42: 09:37:56 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:44:27.414003 | controller |       watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.414009 | controller |     - '    logger.go:42: 09:37:57 | watcher-tls/3-disable-podlevel-tls | + ''['' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.414015 | controller |       == e0a908c724554093bb8de86515131a2c '']'''
2026-03-14 09:44:27.414021 | controller |     - '    logger.go:42: 09:37:57 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:44:27.414026 | controller |       watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.414032 | controller |     - '    logger.go:42: 09:37:57 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.414038 | controller |       '']'''
2026-03-14 09:44:27.414045 | controller |     - '    logger.go:42: 09:37:57 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:44:27.414051 | controller |       watcher-kuttl-default openstackclient -- openstack endpoint list'
2026-03-14 09:44:27.414056 | controller |     - '    logger.go:42: 09:37:57 | watcher-tls/3-disable-podlevel-tls | + grep internal'
2026-03-14 09:44:27.414062 | controller |     - '    logger.go:42: 09:37:57 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim'
2026-03-14 09:44:27.414068 | controller |     - '    logger.go:42: 09:37:57 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https'
2026-03-14 09:44:27.414074 | controller |     - '    logger.go:42: 09:37:59 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:44:27.414084 | controller |       0 '']'''
2026-03-14 09:44:27.414091 | controller |     - '    logger.go:42: 09:38:00 | watcher-tls/3-disable-podlevel-tls | running command:
2026-03-14 09:44:27.414096 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.414102 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.414108 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.414114 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.414120 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.414126 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.414132 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.414148 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.414172 | controller |       ]'
2026-03-14 09:44:27.414179 | controller |     - '        # check that watcher internal endpoint does not use https'
2026-03-14 09:44:27.414184 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.414190 | controller |       infra-optim | grep internal  | [ $(grep -c https) == 0 ]'
2026-03-14 09:44:27.414196 | controller |     - '        # check that watcher public endpoint does use https'
2026-03-14 09:44:27.414202 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.414208 | controller |       infra-optim | grep public  | [ $(grep -c https) == 1 ]'
2026-03-14 09:44:27.414214 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.414220 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.414226 | controller |       == "" ]; then'
2026-03-14 09:44:27.414231 | controller |     - '            exit 0'
2026-03-14 09:44:27.414237 | controller |     - '        fi'
2026-03-14 09:44:27.414243 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.414249 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.414255 | controller |     - '        counter=0'
2026-03-14 09:44:27.414260 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.414266 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.414272 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.414278 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.414284 | controller |     - '            fi'
2026-03-14 09:44:27.414309 | controller |     - '        done'
2026-03-14 09:44:27.414315 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.414321 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.414327 | controller |     - '          exit 1'
2026-03-14 09:44:27.414333 | controller |     - '        else'
2026-03-14 09:44:27.414339 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.414344 | controller |     - '        fi'
2026-03-14 09:44:27.414350 | controller |     - '        ]'
2026-03-14 09:44:27.414360 | controller |     - '    logger.go:42: 09:38:00 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:44:27.414366 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.414371 | controller |       -c Type'
2026-03-14 09:44:27.414378 | controller |     - '    logger.go:42: 09:38:00 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.414383 | controller |     - '    logger.go:42: 09:38:02 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:44:27.414389 | controller |       1 '']'''
2026-03-14 09:44:27.414395 | controller |     - '    logger.go:42: 09:38:02 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher'
2026-03-14 09:44:27.414400 | controller |     - '    logger.go:42: 09:38:02 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print
2026-03-14 09:44:27.414409 | controller |       $1}'''
2026-03-14 09:44:27.414414 | controller |     - '    logger.go:42: 09:38:02 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n
2026-03-14 09:44:27.414420 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.414426 | controller |       -c Type -c ID'
2026-03-14 09:44:27.414432 | controller |     - '    logger.go:42: 09:38:04 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=e0a908c724554093bb8de86515131a2c'
2026-03-14 09:44:27.414437 | controller |     - '    logger.go:42: 09:38:04 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:44:27.414443 | controller |       watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.414449 | controller |     - '    logger.go:42: 09:38:05 | watcher-tls/3-disable-podlevel-tls | + ''['' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.414455 | controller |       == e0a908c724554093bb8de86515131a2c '']'''
2026-03-14 09:44:27.414461 | controller |     - '    logger.go:42: 09:38:05 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:44:27.414467 | controller |       watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.414473 | controller |     - '    logger.go:42: 09:38:05 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.414479 | controller |       '']'''
2026-03-14 09:44:27.414484 | controller |     - '    logger.go:42: 09:38:05 | watcher-tls/3-disable-podlevel-tls | + grep internal'
2026-03-14 09:44:27.414490 | controller |     - '    logger.go:42: 09:38:05 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim'
2026-03-14 09:44:27.414496 | controller |     - '    logger.go:42: 09:38:05 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:44:27.414502 | controller |       watcher-kuttl-default openstackclient -- openstack endpoint list'
2026-03-14 09:44:27.414507 | controller |     - '    logger.go:42: 09:38:05 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https'
2026-03-14 09:44:27.414512 | controller |     - '    logger.go:42: 09:38:07 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:44:27.414518 | controller |       0 '']'''
2026-03-14 09:44:27.414524 | controller |     - '    logger.go:42: 09:38:08 | watcher-tls/3-disable-podlevel-tls | running command:
2026-03-14 09:44:27.414529 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.414539 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.414545 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.414551 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.414557 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.414570 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.414576 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.414581 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.414586 | controller |       ]'
2026-03-14 09:44:27.414592 | controller |     - '        # check that watcher internal endpoint does not use https'
2026-03-14 09:44:27.414597 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.414602 | controller |       infra-optim | grep internal  | [ $(grep -c https) == 0 ]'
2026-03-14 09:44:27.414607 | controller |     - '        # check that watcher public endpoint does use https'
2026-03-14 09:44:27.414612 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.414618 | controller |       infra-optim | grep public  | [ $(grep -c https) == 1 ]'
2026-03-14 09:44:27.414624 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.414630 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.414635 | controller |       == "" ]; then'
2026-03-14 09:44:27.414640 | controller |     - '            exit 0'
2026-03-14 09:44:27.414649 | controller |     - '        fi'
2026-03-14 09:44:27.414655 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.414661 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.414666 | controller |     - '        counter=0'
2026-03-14 09:44:27.414672 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.414678 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.414683 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.414689 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.414695 | controller |     - '            fi'
2026-03-14 09:44:27.414701 | controller |     - '        done'
2026-03-14 09:44:27.414706 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.414712 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.414718 | controller |     - '          exit 1'
2026-03-14 09:44:27.414724 | controller |     - '        else'
2026-03-14 09:44:27.414730 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.414735 | controller |     - '        fi'
2026-03-14 09:44:27.414741 | controller |     - '        ]'
2026-03-14 09:44:27.414747 | controller |     - '    logger.go:42: 09:38:08 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:44:27.414753 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.414759 | controller |       -c Type'
2026-03-14 09:44:27.414764 | controller |     - '    logger.go:42: 09:38:08 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.414770 | controller |     - '    logger.go:42: 09:38:10 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:44:27.414776 | controller |       1 '']'''
2026-03-14 09:44:27.414782 | controller |     - '    logger.go:42: 09:38:10 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print
2026-03-14 09:44:27.414787 | controller |       $1}'''
2026-03-14 09:44:27.414793 | controller |     - '    logger.go:42: 09:38:10 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher'
2026-03-14 09:44:27.414799 | controller |     - '    logger.go:42: 09:38:10 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n
2026-03-14 09:44:27.414805 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.414811 | controller |       -c Type -c ID'
2026-03-14 09:44:27.414817 | controller |     - '    logger.go:42: 09:38:12 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=e0a908c724554093bb8de86515131a2c'
2026-03-14 09:44:27.414823 | controller |     - '    logger.go:42: 09:38:12 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:44:27.414829 | controller |       watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.414835 | controller |     - '    logger.go:42: 09:38:12 | watcher-tls/3-disable-podlevel-tls | + ''['' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.414840 | controller |       == e0a908c724554093bb8de86515131a2c '']'''
2026-03-14 09:44:27.414846 | controller |     - '    logger.go:42: 09:38:12 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:44:27.414852 | controller |       watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.414858 | controller |     - '    logger.go:42: 09:38:12 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.414864 | controller |       '']'''
2026-03-14 09:44:27.414870 | controller |     - '    logger.go:42: 09:38:12 | watcher-tls/3-disable-podlevel-tls | + grep internal'
2026-03-14 09:44:27.414875 | controller |     - '    logger.go:42: 09:38:12 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim'
2026-03-14 09:44:27.414881 | controller |     - '    logger.go:42: 09:38:12 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:44:27.414887 | controller |       watcher-kuttl-default openstackclient -- openstack endpoint list'
2026-03-14 09:44:27.414893 | controller |     - '    logger.go:42: 09:38:12 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https'
2026-03-14 09:44:27.414899 | controller |     - '    logger.go:42: 09:38:14 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:44:27.414908 | controller |       0 '']'''
2026-03-14 09:44:27.414915 | controller |     - '    logger.go:42: 09:38:15 | watcher-tls/3-disable-podlevel-tls | running command:
2026-03-14 09:44:27.414921 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.414929 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.414935 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.414951 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.414958 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.414964 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.414969 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.414975 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.414981 | controller |       ]'
2026-03-14 09:44:27.414987 | controller |     - '        # check that watcher internal endpoint does not use https'
2026-03-14 09:44:27.414993 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.414999 | controller |       infra-optim | grep internal  | [ $(grep -c https) == 0 ]'
2026-03-14 09:44:27.415005 | controller |     - '        # check that watcher public endpoint does use https'
2026-03-14 09:44:27.415011 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.415016 | controller |       infra-optim | grep public  | [ $(grep -c https) == 1 ]'
2026-03-14 09:44:27.415022 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.415028 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.415033 | controller |       == "" ]; then'
2026-03-14 09:44:27.415039 | controller |     - '            exit 0'
2026-03-14 09:44:27.415045 | controller |     - '        fi'
2026-03-14 09:44:27.415051 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.415057 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.415063 | controller |     - '        counter=0'
2026-03-14 09:44:27.415069 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.415079 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.415085 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.415091 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.415097 | controller |     - '            fi'
2026-03-14 09:44:27.415103 | controller |     - '        done'
2026-03-14 09:44:27.415108 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.415114 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.415120 | controller |     - '          exit 1'
2026-03-14 09:44:27.415126 | controller |     - '        else'
2026-03-14 09:44:27.415132 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.415137 | controller |     - '        fi'
2026-03-14 09:44:27.415143 | controller |     - '        ]'
2026-03-14 09:44:27.415149 | controller |     - '    logger.go:42: 09:38:15 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:44:27.415155 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.415161 | controller |       -c Type'
2026-03-14 09:44:27.415167 | controller |     - '    logger.go:42: 09:38:15 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.415173 | controller |     - '    logger.go:42: 09:38:17 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:44:27.415179 | controller |       1 '']'''
2026-03-14 09:44:27.415185 | controller |     - '    logger.go:42: 09:38:17 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher'
2026-03-14 09:44:27.415193 | controller |     - '    logger.go:42: 09:38:17 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n
2026-03-14 09:44:27.415199 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.415207 | controller |       -c Type -c ID'
2026-03-14 09:44:27.415213 | controller |     - '    logger.go:42: 09:38:17 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print
2026-03-14 09:44:27.415219 | controller |       $1}'''
2026-03-14 09:44:27.415225 | controller |     - '    logger.go:42: 09:38:19 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=e0a908c724554093bb8de86515131a2c'
2026-03-14 09:44:27.415231 | controller |     - '    logger.go:42: 09:38:19 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:44:27.415237 | controller |       watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.415242 | controller |     - '    logger.go:42: 09:38:19 | watcher-tls/3-disable-podlevel-tls | + ''['' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.415248 | controller |       == e0a908c724554093bb8de86515131a2c '']'''
2026-03-14 09:44:27.415255 | controller |     - '    logger.go:42: 09:38:19 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:44:27.415260 | controller |       watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.415266 | controller |     - '    logger.go:42: 09:38:19 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.415272 | controller |       '']'''
2026-03-14 09:44:27.415278 | controller |     - '    logger.go:42: 09:38:19 | watcher-tls/3-disable-podlevel-tls | + grep internal'
2026-03-14 09:44:27.415284 | controller |     - '    logger.go:42: 09:38:19 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim'
2026-03-14 09:44:27.415306 | controller |     - '    logger.go:42: 09:38:19 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:44:27.415316 | controller |       watcher-kuttl-default openstackclient -- openstack endpoint list'
2026-03-14 09:44:27.415322 | controller |     - '    logger.go:42: 09:38:19 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https'
2026-03-14 09:44:27.415328 | controller |     - '    logger.go:42: 09:38:21 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:44:27.415334 | controller |       0 '']'''
2026-03-14 09:44:27.415340 | controller |     - '    logger.go:42: 09:38:22 | watcher-tls/3-disable-podlevel-tls | running command:
2026-03-14 09:44:27.415345 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.415360 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.415367 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.415373 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.415379 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.415384 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.415390 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.415396 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.415402 | controller |       ]'
2026-03-14 09:44:27.415408 | controller |     - '        # check that watcher internal endpoint does not use https'
2026-03-14 09:44:27.415414 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.415420 | controller |       infra-optim | grep internal  | [ $(grep -c https) == 0 ]'
2026-03-14 09:44:27.415426 | controller |     - '        # check that watcher public endpoint does use https'
2026-03-14 09:44:27.415432 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.415438 | controller |       infra-optim | grep public  | [ $(grep -c https) == 1 ]'
2026-03-14 09:44:27.415443 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.415449 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.415455 | controller |       == "" ]; then'
2026-03-14 09:44:27.415461 | controller |     - '            exit 0'
2026-03-14 09:44:27.415466 | controller |     - '        fi'
2026-03-14 09:44:27.415476 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.415482 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.415487 | controller |     - '        counter=0'
2026-03-14 09:44:27.415494 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.415500 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.415506 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.415512 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.415518 | controller |     - '            fi'
2026-03-14 09:44:27.415523 | controller |     - '        done'
2026-03-14 09:44:27.415529 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.415535 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.415541 | controller |     - '          exit 1'
2026-03-14 09:44:27.415547 | controller |     - '        else'
2026-03-14 09:44:27.415552 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.415558 | controller |     - '        fi'
2026-03-14 09:44:27.415564 | controller |     - '        ]'
2026-03-14 09:44:27.415570 | controller |     - '    logger.go:42: 09:38:22 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:44:27.415576 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.415582 | controller |       -c Type'
2026-03-14 09:44:27.415588 | controller |     - '    logger.go:42: 09:38:22 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.415594 | controller |     - '    logger.go:42: 09:38:26 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:44:27.415600 | controller |       1 '']'''
2026-03-14 09:44:27.415606 | controller |     - '    logger.go:42: 09:38:26 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher'
2026-03-14 09:44:27.415611 | controller |     - '    logger.go:42: 09:38:26 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print
2026-03-14 09:44:27.415617 | controller |       $1}'''
2026-03-14 09:44:27.415623 | controller |     - '    logger.go:42: 09:38:26 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n
2026-03-14 09:44:27.415629 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.415635 | controller |       -c Type -c ID'
2026-03-14 09:44:27.415640 | controller |     - '    logger.go:42: 09:38:29 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=e0a908c724554093bb8de86515131a2c'
2026-03-14 09:44:27.415646 | controller |     - '    logger.go:42: 09:38:29 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:44:27.415652 | controller |       watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.415658 | controller |     - '    logger.go:42: 09:38:30 | watcher-tls/3-disable-podlevel-tls | + ''['' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.415664 | controller |       == e0a908c724554093bb8de86515131a2c '']'''
2026-03-14 09:44:27.415670 | controller |     - '    logger.go:42: 09:38:30 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:44:27.415676 | controller |       watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.415685 | controller |     - '    logger.go:42: 09:38:31 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.415691 | controller |       '']'''
2026-03-14 09:44:27.415697 | controller |     - '    logger.go:42: 09:38:31 | watcher-tls/3-disable-podlevel-tls | + grep internal'
2026-03-14 09:44:27.415703 | controller |     - '    logger.go:42: 09:38:31 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https'
2026-03-14 09:44:27.415710 | controller |     - '    logger.go:42: 09:38:31 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim'
2026-03-14 09:44:27.415716 | controller |     - '    logger.go:42: 09:38:31 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:44:27.415722 | controller |       watcher-kuttl-default openstackclient -- openstack endpoint list'
2026-03-14 09:44:27.415727 | controller |     - '    logger.go:42: 09:38:33 | watcher-tls/3-disable-podlevel-tls | + ''['' 0 ==
2026-03-14 09:44:27.415736 | controller |       0 '']'''
2026-03-14 09:44:27.415750 | controller |     - '    logger.go:42: 09:38:33 | watcher-tls/3-disable-podlevel-tls | + grep public'
2026-03-14 09:44:27.415757 | controller |     - '    logger.go:42: 09:38:33 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim'
2026-03-14 09:44:27.415763 | controller |     - '    logger.go:42: 09:38:33 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:44:27.415769 | controller |       watcher-kuttl-default openstackclient -- openstack endpoint list'
2026-03-14 09:44:27.415775 | controller |     - '    logger.go:42: 09:38:33 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https'
2026-03-14 09:44:27.415780 | controller |     - '    logger.go:42: 09:38:35 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:44:27.415786 | controller |       1 '']'''
2026-03-14 09:44:27.415792 | controller |     - '    logger.go:42: 09:38:35 | watcher-tls/3-disable-podlevel-tls | ++ oc get pods
2026-03-14 09:44:27.415798 | controller |       -n openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:44:27.415804 | controller |     - '    logger.go:42: 09:38:35 | watcher-tls/3-disable-podlevel-tls | + ''['' ''''
2026-03-14 09:44:27.415809 | controller |       == '''' '']'''
2026-03-14 09:44:27.415815 | controller |     - '    logger.go:42: 09:38:35 | watcher-tls/3-disable-podlevel-tls | + exit 0'
2026-03-14 09:44:27.415821 | controller |     - '    logger.go:42: 09:38:36 | watcher-tls/3-disable-podlevel-tls | running command:
2026-03-14 09:44:27.415827 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.415833 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.415840 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.415846 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.415852 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.415858 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.415863 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.415869 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.415875 | controller |       ]'
2026-03-14 09:44:27.415881 | controller |     - '        # check that watcher internal endpoint does not use https'
2026-03-14 09:44:27.415887 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.415892 | controller |       infra-optim | grep internal  | [ $(grep -c https) == 0 ]'
2026-03-14 09:44:27.415898 | controller |     - '        # check that watcher public endpoint does use https'
2026-03-14 09:44:27.415904 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.415910 | controller |       infra-optim | grep public  | [ $(grep -c https) == 1 ]'
2026-03-14 09:44:27.415916 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.415922 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.415929 | controller |       == "" ]; then'
2026-03-14 09:44:27.415935 | controller |     - '            exit 0'
2026-03-14 09:44:27.415940 | controller |     - '        fi'
2026-03-14 09:44:27.415946 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.415953 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.415958 | controller |     - '        counter=0'
2026-03-14 09:44:27.415966 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.415973 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.415979 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.415986 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.415993 | controller |     - '            fi'
2026-03-14 09:44:27.415999 | controller |     - '        done'
2026-03-14 09:44:27.416005 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.416011 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.416020 | controller |     - '          exit 1'
2026-03-14 09:44:27.416026 | controller |     - '        else'
2026-03-14 09:44:27.416032 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.416037 | controller |     - '        fi'
2026-03-14 09:44:27.416043 | controller |     - '        ]'
2026-03-14 09:44:27.416049 | controller |     - '    logger.go:42: 09:38:36 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:44:27.416055 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.416061 | controller |       -c Type'
2026-03-14 09:44:27.416066 | controller |     - '    logger.go:42: 09:38:36 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.416072 | controller |     - '    logger.go:42: 09:38:38 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:44:27.416078 | controller |       1 '']'''
2026-03-14 09:44:27.416084 | controller |     - '    logger.go:42: 09:38:38 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n
2026-03-14 09:44:27.416090 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.416096 | controller |       -c Type -c ID'
2026-03-14 09:44:27.416102 | controller |     - '    logger.go:42: 09:38:38 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print
2026-03-14 09:44:27.416107 | controller |       $1}'''
2026-03-14 09:44:27.416113 | controller |     - '    logger.go:42: 09:38:38 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher'
2026-03-14 09:44:27.416119 | controller |     - '    logger.go:42: 09:38:40 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=e0a908c724554093bb8de86515131a2c'
2026-03-14 09:44:27.416125 | controller |     - '    logger.go:42: 09:38:40 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:44:27.416131 | controller |       watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.416144 | controller |     - '    logger.go:42: 09:38:40 | watcher-tls/3-disable-podlevel-tls | + ''['' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.416150 | controller |       == e0a908c724554093bb8de86515131a2c '']'''
2026-03-14 09:44:27.416156 | controller |     - '    logger.go:42: 09:38:40 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n
2026-03-14 09:44:27.416162 | controller |       watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.416168 | controller |     - '    logger.go:42: 09:38:40 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.416174 | controller |       '']'''
2026-03-14 09:44:27.416180 | controller |     - '    logger.go:42: 09:38:40 | watcher-tls/3-disable-podlevel-tls | + grep internal'
2026-03-14 09:44:27.416186 | controller |     - '    logger.go:42: 09:38:40 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim'
2026-03-14 09:44:27.416192 | controller |     - '    logger.go:42: 09:38:40 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https'
2026-03-14 09:44:27.416201 | controller |     - '    logger.go:42: 09:38:40 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:44:27.416207 | controller |       watcher-kuttl-default openstackclient -- openstack endpoint list'
2026-03-14 09:44:27.416213 | controller |     - '    logger.go:42: 09:38:42 | watcher-tls/3-disable-podlevel-tls | + ''['' 0 ==
2026-03-14 09:44:27.416219 | controller |       0 '']'''
2026-03-14 09:44:27.416224 | controller |     - '    logger.go:42: 09:38:42 | watcher-tls/3-disable-podlevel-tls | + oc exec -n
2026-03-14 09:44:27.416230 | controller |       watcher-kuttl-default openstackclient -- openstack endpoint list'
2026-03-14 09:44:27.416236 | controller |     - '    logger.go:42: 09:38:42 | watcher-tls/3-disable-podlevel-tls | + grep public'
2026-03-14 09:44:27.416242 | controller |     - '    logger.go:42: 09:38:42 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim'
2026-03-14 09:44:27.416247 | controller |     - '    logger.go:42: 09:38:42 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https'
2026-03-14 09:44:27.416253 | controller |     - '    logger.go:42: 09:38:45 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 ==
2026-03-14 09:44:27.416259 | controller |       1 '']'''
2026-03-14 09:44:27.416265 | controller |     - '    logger.go:42: 09:38:45 | watcher-tls/3-disable-podlevel-tls | ++ oc get pods
2026-03-14 09:44:27.416276 | controller |       -n openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:44:27.416282 | controller |     - '    logger.go:42: 09:38:45 | watcher-tls/3-disable-podlevel-tls | + ''['' ''''
2026-03-14 09:44:27.416291 | controller |       == '''' '']'''
2026-03-14 09:44:27.416313 | controller |     - '    logger.go:42: 09:38:45 | watcher-tls/3-disable-podlevel-tls | + exit 0'
2026-03-14 09:44:27.416319 | controller |     - '    logger.go:42: 09:38:45 | watcher-tls/3-disable-podlevel-tls | test step completed
2026-03-14 09:44:27.416324 | controller |       3-disable-podlevel-tls'
2026-03-14 09:44:27.416330 | controller |     - '    logger.go:42: 09:38:45 | watcher-tls/4-deploy-without-route | starting test
2026-03-14 09:44:27.416336 | controller |       step 4-deploy-without-route'
2026-03-14 09:44:27.416342 | controller |     - '    logger.go:42: 09:38:45 | watcher-tls/4-deploy-without-route | running command:
2026-03-14 09:44:27.416348 | controller |       [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op":
2026-03-14 09:44:27.416354 | controller |       "replace", "path": "/spec/apiServiceTemplate/override", "value":{"service": { "internal":
2026-03-14 09:44:27.416359 | controller |       {}, "public": { "metadata": { "annotations": { "metallb.universe.tf/address-pool":
2026-03-14 09:44:27.416366 | controller |       "ctlplane", "metallb.universe.tf/allow-shared-ip": "ctlplane" } }, "spec": { "type":
2026-03-14 09:44:27.416372 | controller |       "LoadBalancer" } } } }}]'''
2026-03-14 09:44:27.416378 | controller |     - '        ]'
2026-03-14 09:44:27.416383 | controller |     - '    logger.go:42: 09:38:45 | watcher-tls/4-deploy-without-route | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:44:27.416389 | controller |       patched'
2026-03-14 09:44:27.416395 | controller |     - '    logger.go:42: 09:38:45 | watcher-tls/4-deploy-without-route | running command:
2026-03-14 09:44:27.416401 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.416406 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.416412 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.416418 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.416424 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.416429 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.416435 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.416442 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.416448 | controller |       ]'
2026-03-14 09:44:27.416454 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.416460 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.416466 | controller |       == "" ]; then'
2026-03-14 09:44:27.416472 | controller |     - '            exit 0'
2026-03-14 09:44:27.416478 | controller |     - '        fi'
2026-03-14 09:44:27.416483 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.416489 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.416499 | controller |     - '        counter=0'
2026-03-14 09:44:27.416505 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.416511 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.416517 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.416523 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.416540 | controller |     - '            fi'
2026-03-14 09:44:27.416547 | controller |     - '        done'
2026-03-14 09:44:27.416553 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.416559 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.416565 | controller |     - '          exit 1'
2026-03-14 09:44:27.416574 | controller |     - '        else'
2026-03-14 09:44:27.416580 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.416589 | controller |     - '        fi'
2026-03-14 09:44:27.416595 | controller |     - '        ]'
2026-03-14 09:44:27.416601 | controller |     - '    logger.go:42: 09:38:45 | watcher-tls/4-deploy-without-route | + oc exec -n
2026-03-14 09:44:27.416608 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.416614 | controller |       -c Type'
2026-03-14 09:44:27.416623 | controller |     - '    logger.go:42: 09:38:45 | watcher-tls/4-deploy-without-route | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.416629 | controller |     - '    logger.go:42: 09:38:47 | watcher-tls/4-deploy-without-route | + ''['' 1 ==
2026-03-14 09:44:27.416635 | controller |       1 '']'''
2026-03-14 09:44:27.416641 | controller |     - '    logger.go:42: 09:38:47 | watcher-tls/4-deploy-without-route | ++ grep watcher'
2026-03-14 09:44:27.416647 | controller |     - '    logger.go:42: 09:38:47 | watcher-tls/4-deploy-without-route | ++ awk ''{print
2026-03-14 09:44:27.416653 | controller |       $1}'''
2026-03-14 09:44:27.416659 | controller |     - '    logger.go:42: 09:38:47 | watcher-tls/4-deploy-without-route | ++ oc exec -n
2026-03-14 09:44:27.416665 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.416670 | controller |       -c Type -c ID'
2026-03-14 09:44:27.416676 | controller |     - '    logger.go:42: 09:38:49 | watcher-tls/4-deploy-without-route | + SERVICEID=e0a908c724554093bb8de86515131a2c'
2026-03-14 09:44:27.416682 | controller |     - '    logger.go:42: 09:38:49 | watcher-tls/4-deploy-without-route | ++ oc get -n
2026-03-14 09:44:27.416688 | controller |       watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.416694 | controller |     - '    logger.go:42: 09:38:49 | watcher-tls/4-deploy-without-route | + ''['' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.416700 | controller |       == e0a908c724554093bb8de86515131a2c '']'''
2026-03-14 09:44:27.416705 | controller |     - '    logger.go:42: 09:38:49 | watcher-tls/4-deploy-without-route | ++ oc get -n
2026-03-14 09:44:27.416712 | controller |       watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.416718 | controller |     - '    logger.go:42: 09:38:50 | watcher-tls/4-deploy-without-route | + ''['' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.416724 | controller |       '']'''
2026-03-14 09:44:27.416733 | controller |     - '    logger.go:42: 09:38:50 | watcher-tls/4-deploy-without-route | ++ oc get pods
2026-03-14 09:44:27.416739 | controller |       -n openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:44:27.416745 | controller |     - '    logger.go:42: 09:38:50 | watcher-tls/4-deploy-without-route | + ''['' ''''
2026-03-14 09:44:27.416751 | controller |       == '''' '']'''
2026-03-14 09:44:27.416757 | controller |     - '    logger.go:42: 09:38:50 | watcher-tls/4-deploy-without-route | + exit 0'
2026-03-14 09:44:27.416762 | controller |     - '    logger.go:42: 09:38:51 | watcher-tls/4-deploy-without-route | running command:
2026-03-14 09:44:27.416768 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.416774 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.416780 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.416786 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.416792 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.416798 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.416804 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.416811 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.416816 | controller |       ]'
2026-03-14 09:44:27.416825 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.416831 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.416839 | controller |       == "" ]; then'
2026-03-14 09:44:27.416845 | controller |     - '            exit 0'
2026-03-14 09:44:27.416851 | controller |     - '        fi'
2026-03-14 09:44:27.416857 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.416862 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.416868 | controller |     - '        counter=0'
2026-03-14 09:44:27.416874 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.416880 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.416886 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.416892 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.416898 | controller |     - '            fi'
2026-03-14 09:44:27.416904 | controller |     - '        done'
2026-03-14 09:44:27.416910 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.416916 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.416921 | controller |     - '          exit 1'
2026-03-14 09:44:27.416927 | controller |     - '        else'
2026-03-14 09:44:27.416933 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.416939 | controller |     - '        fi'
2026-03-14 09:44:27.416944 | controller |     - '        ]'
2026-03-14 09:44:27.416950 | controller |     - '    logger.go:42: 09:38:51 | watcher-tls/4-deploy-without-route | + oc exec -n
2026-03-14 09:44:27.416956 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.416972 | controller |       -c Type'
2026-03-14 09:44:27.416979 | controller |     - '    logger.go:42: 09:38:51 | watcher-tls/4-deploy-without-route | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.416985 | controller |     - '    logger.go:42: 09:38:53 | watcher-tls/4-deploy-without-route | + ''['' 1 ==
2026-03-14 09:44:27.416991 | controller |       1 '']'''
2026-03-14 09:44:27.416996 | controller |     - '    logger.go:42: 09:38:53 | watcher-tls/4-deploy-without-route | ++ grep watcher'
2026-03-14 09:44:27.417002 | controller |     - '    logger.go:42: 09:38:53 | watcher-tls/4-deploy-without-route | ++ awk ''{print
2026-03-14 09:44:27.417008 | controller |       $1}'''
2026-03-14 09:44:27.417014 | controller |     - '    logger.go:42: 09:38:53 | watcher-tls/4-deploy-without-route | ++ oc exec -n
2026-03-14 09:44:27.417020 | controller |       watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.417026 | controller |       -c Type -c ID'
2026-03-14 09:44:27.417031 | controller |     - '    logger.go:42: 09:38:55 | watcher-tls/4-deploy-without-route | + SERVICEID=e0a908c724554093bb8de86515131a2c'
2026-03-14 09:44:27.417037 | controller |     - '    logger.go:42: 09:38:55 | watcher-tls/4-deploy-without-route | ++ oc get -n
2026-03-14 09:44:27.417046 | controller |       watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.417052 | controller |     - '    logger.go:42: 09:38:55 | watcher-tls/4-deploy-without-route | + ''['' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.417059 | controller |       == e0a908c724554093bb8de86515131a2c '']'''
2026-03-14 09:44:27.417064 | controller |     - '    logger.go:42: 09:38:55 | watcher-tls/4-deploy-without-route | ++ oc get -n
2026-03-14 09:44:27.417070 | controller |       watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.417076 | controller |     - '    logger.go:42: 09:38:55 | watcher-tls/4-deploy-without-route | + ''['' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.417082 | controller |       '']'''
2026-03-14 09:44:27.417088 | controller |     - '    logger.go:42: 09:38:55 | watcher-tls/4-deploy-without-route | ++ oc get pods
2026-03-14 09:44:27.417093 | controller |       -n openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:44:27.417099 | controller |     - '    logger.go:42: 09:38:55 | watcher-tls/4-deploy-without-route | + ''['' ''''
2026-03-14 09:44:27.417105 | controller |       == '''' '']'''
2026-03-14 09:44:27.417111 | controller |     - '    logger.go:42: 09:38:55 | watcher-tls/4-deploy-without-route | + exit 0'
2026-03-14 09:44:27.417117 | controller |     - '    logger.go:42: 09:38:55 | watcher-tls/4-deploy-without-route | test step completed
2026-03-14 09:44:27.417126 | controller |       4-deploy-without-route'
2026-03-14 09:44:27.417132 | controller |     - '    logger.go:42: 09:38:55 | watcher-tls/5-disable-tls | starting test step 5-disable-tls'
2026-03-14 09:44:27.417138 | controller |     - '    logger.go:42: 09:38:55 | watcher-tls/5-disable-tls | running command: [sh -c
2026-03-14 09:44:27.417144 | controller |       oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op": "replace",
2026-03-14 09:44:27.417149 | controller |       "path": "/spec/apiServiceTemplate/override", "value":{}}]'''
2026-03-14 09:44:27.417155 | controller |     - '        ]'
2026-03-14 09:44:27.417161 | controller |     - '    logger.go:42: 09:38:55 | watcher-tls/5-disable-tls | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:44:27.417167 | controller |       patched'
2026-03-14 09:44:27.417173 | controller |     - '    logger.go:42: 09:38:55 | watcher-tls/5-disable-tls | running command: [sh -c
2026-03-14 09:44:27.417179 | controller |       set -euxo pipefail'
2026-03-14 09:44:27.417184 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.417190 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.417196 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.417202 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.417207 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.417213 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.417219 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.417225 | controller |       ]'
2026-03-14 09:44:27.417231 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.417237 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.417243 | controller |       == "" ]; then'
2026-03-14 09:44:27.417248 | controller |     - '            exit 0'
2026-03-14 09:44:27.417254 | controller |     - '        fi'
2026-03-14 09:44:27.417260 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.417266 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.417272 | controller |     - '        counter=0'
2026-03-14 09:44:27.417277 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.417283 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.417305 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.417316 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.417322 | controller |     - '            fi'
2026-03-14 09:44:27.417328 | controller |     - '        done'
2026-03-14 09:44:27.417333 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.417339 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.417345 | controller |     - '          exit 1'
2026-03-14 09:44:27.417351 | controller |     - '        else'
2026-03-14 09:44:27.417357 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.417362 | controller |     - '        fi'
2026-03-14 09:44:27.417368 | controller |     - '        # check that no watcher endpoint uses https'
2026-03-14 09:44:27.417383 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.417390 | controller |       infra-optim | [ $(grep -c https) == 0 ]'
2026-03-14 09:44:27.417396 | controller |     - '        ]'
2026-03-14 09:44:27.417402 | controller |     - '    logger.go:42: 09:38:55 | watcher-tls/5-disable-tls | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.417407 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.417413 | controller |     - '    logger.go:42: 09:38:55 | watcher-tls/5-disable-tls | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.417419 | controller |     - '    logger.go:42: 09:38:57 | watcher-tls/5-disable-tls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.417431 | controller |     - '    logger.go:42: 09:38:57 | watcher-tls/5-disable-tls | ++ grep watcher'
2026-03-14 09:44:27.417437 | controller |     - '    logger.go:42: 09:38:57 | watcher-tls/5-disable-tls | ++ awk ''{print $1}'''
2026-03-14 09:44:27.417442 | controller |     - '    logger.go:42: 09:38:57 | watcher-tls/5-disable-tls | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.417448 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:44:27.417454 | controller |     - '    logger.go:42: 09:38:59 | watcher-tls/5-disable-tls | + SERVICEID=e0a908c724554093bb8de86515131a2c'
2026-03-14 09:44:27.417460 | controller |     - '    logger.go:42: 09:38:59 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.417466 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.417472 | controller |     - '    logger.go:42: 09:38:59 | watcher-tls/5-disable-tls | + ''['' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.417478 | controller |       == e0a908c724554093bb8de86515131a2c '']'''
2026-03-14 09:44:27.417484 | controller |     - '    logger.go:42: 09:38:59 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.417489 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.417495 | controller |     - '    logger.go:42: 09:39:00 | watcher-tls/5-disable-tls | + ''['' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.417502 | controller |       '']'''
2026-03-14 09:44:27.417507 | controller |     - '    logger.go:42: 09:39:00 | watcher-tls/5-disable-tls | ++ oc get pods -n openstack-operators
2026-03-14 09:44:27.417513 | controller |       -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:44:27.417519 | controller |     - '    logger.go:42: 09:39:00 | watcher-tls/5-disable-tls | + ''['' '''' == '''' '']'''
2026-03-14 09:44:27.417525 | controller |     - '    logger.go:42: 09:39:00 | watcher-tls/5-disable-tls | + exit 0'
2026-03-14 09:44:27.417531 | controller |     - '    logger.go:42: 09:39:01 | watcher-tls/5-disable-tls | running command: [sh -c
2026-03-14 09:44:27.417536 | controller |       set -euxo pipefail'
2026-03-14 09:44:27.417542 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value
2026-03-14 09:44:27.417548 | controller |       -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.417554 | controller |     - '        SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service
2026-03-14 09:44:27.417560 | controller |       list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.417566 | controller |     - '        [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.417572 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.417578 | controller |     - '        [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.417583 | controller |       ]'
2026-03-14 09:44:27.417589 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.417595 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.417601 | controller |       == "" ]; then'
2026-03-14 09:44:27.417606 | controller |     - '            exit 0'
2026-03-14 09:44:27.417612 | controller |     - '        fi'
2026-03-14 09:44:27.417618 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.417623 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.417630 | controller |     - '        counter=0'
2026-03-14 09:44:27.417636 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.417642 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.417647 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.417653 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.417659 | controller |     - '            fi'
2026-03-14 09:44:27.417665 | controller |     - '        done'
2026-03-14 09:44:27.417671 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.417679 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.417685 | controller |     - '          exit 1'
2026-03-14 09:44:27.417691 | controller |     - '        else'
2026-03-14 09:44:27.417697 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.417703 | controller |     - '        fi'
2026-03-14 09:44:27.417709 | controller |     - '        # check that no watcher endpoint uses https'
2026-03-14 09:44:27.417715 | controller |     - '        oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep
2026-03-14 09:44:27.417721 | controller |       infra-optim | [ $(grep -c https) == 0 ]'
2026-03-14 09:44:27.417726 | controller |     - '        ]'
2026-03-14 09:44:27.417732 | controller |     - '    logger.go:42: 09:39:01 | watcher-tls/5-disable-tls | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.417738 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.417744 | controller |     - '    logger.go:42: 09:39:01 | watcher-tls/5-disable-tls | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.417758 | controller |     - '    logger.go:42: 09:39:03 | watcher-tls/5-disable-tls | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.417764 | controller |     - '    logger.go:42: 09:39:03 | watcher-tls/5-disable-tls | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.417770 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:44:27.417776 | controller |     - '    logger.go:42: 09:39:03 | watcher-tls/5-disable-tls | ++ awk ''{print $1}'''
2026-03-14 09:44:27.417782 | controller |     - '    logger.go:42: 09:39:03 | watcher-tls/5-disable-tls | ++ grep watcher'
2026-03-14 09:44:27.417788 | controller |     - '    logger.go:42: 09:39:05 | watcher-tls/5-disable-tls | + SERVICEID=e0a908c724554093bb8de86515131a2c'
2026-03-14 09:44:27.417794 | controller |     - '    logger.go:42: 09:39:05 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.417800 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.417805 | controller |     - '    logger.go:42: 09:39:05 | watcher-tls/5-disable-tls | + ''['' e0a908c724554093bb8de86515131a2c
2026-03-14 09:44:27.417811 | controller |       == e0a908c724554093bb8de86515131a2c '']'''
2026-03-14 09:44:27.417817 | controller |     - '    logger.go:42: 09:39:05 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.417823 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.417829 | controller |     - '    logger.go:42: 09:39:05 | watcher-tls/5-disable-tls | + ''['' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.417835 | controller |       '']'''
2026-03-14 09:44:27.417840 | controller |     - '    logger.go:42: 09:39:05 | watcher-tls/5-disable-tls | ++ oc get pods -n openstack-operators
2026-03-14 09:44:27.417846 | controller |       -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:44:27.417852 | controller |     - '    logger.go:42: 09:39:05 | watcher-tls/5-disable-tls | + ''['' '''' == '''' '']'''
2026-03-14 09:44:27.417858 | controller |     - '    logger.go:42: 09:39:05 | watcher-tls/5-disable-tls | + exit 0'
2026-03-14 09:44:27.417864 | controller |     - '    logger.go:42: 09:39:05 | watcher-tls/5-disable-tls | test step completed 5-disable-tls'
2026-03-14 09:44:27.417870 | controller |     - '    logger.go:42: 09:39:05 | watcher-tls/6-cleanup-watcher | starting test step
2026-03-14 09:44:27.417876 | controller |       6-cleanup-watcher'
2026-03-14 09:44:27.417882 | controller |     - '    logger.go:42: 09:39:05 | watcher-tls/6-cleanup-watcher | running command: [sh
2026-03-14 09:44:27.417888 | controller |       -c set -ex'
2026-03-14 09:44:27.417893 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.417899 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:44:27.417905 | controller |     - '        ]'
2026-03-14 09:44:27.417914 | controller |     - '    logger.go:42: 09:39:05 | watcher-tls/6-cleanup-watcher | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.417920 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.417926 | controller |     - '    logger.go:42: 09:39:05 | watcher-tls/6-cleanup-watcher | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.417934 | controller |     - '    logger.go:42: 09:39:08 | watcher-tls/6-cleanup-watcher | + ''['' 1 == 0 '']'''
2026-03-14 09:44:27.417940 | controller |     - '    logger.go:42: 09:39:09 | watcher-tls/6-cleanup-watcher | running command: [sh
2026-03-14 09:44:27.417946 | controller |       -c set -ex'
2026-03-14 09:44:27.417952 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.417958 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:44:27.417964 | controller |     - '        ]'
2026-03-14 09:44:27.417971 | controller |     - '    logger.go:42: 09:39:09 | watcher-tls/6-cleanup-watcher | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.417978 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.417984 | controller |     - '    logger.go:42: 09:39:09 | watcher-tls/6-cleanup-watcher | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.417991 | controller |     - '    logger.go:42: 09:39:11 | watcher-tls/6-cleanup-watcher | + ''['' 0 == 0 '']'''
2026-03-14 09:44:27.417998 | controller |     - '    logger.go:42: 09:39:12 | watcher-tls/6-cleanup-watcher | running command: [sh
2026-03-14 09:44:27.418004 | controller |       -c set -ex'
2026-03-14 09:44:27.418010 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.418015 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:44:27.418021 | controller |     - '        ]'
2026-03-14 09:44:27.418027 | controller |     - '    logger.go:42: 09:39:12 | watcher-tls/6-cleanup-watcher | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.418033 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.418039 | controller |     - '    logger.go:42: 09:39:12 | watcher-tls/6-cleanup-watcher | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.418045 | controller |     - '    logger.go:42: 09:39:14 | watcher-tls/6-cleanup-watcher | + ''['' 0 == 0 '']'''
2026-03-14 09:44:27.418051 | controller |     - '    logger.go:42: 09:39:14 | watcher-tls/6-cleanup-watcher | test step completed
2026-03-14 09:44:27.418057 | controller |       6-cleanup-watcher'
2026-03-14 09:44:27.418068 | controller |     - '    logger.go:42: 09:39:14 | watcher-tls/7-cleanup-certs | starting test step 7-cleanup-certs'
2026-03-14 09:44:27.418074 | controller |     - '    logger.go:42: 09:39:14 | watcher-tls/7-cleanup-certs | test step completed
2026-03-14 09:44:27.418080 | controller |       7-cleanup-certs'
2026-03-14 09:44:27.418086 | controller |     - '    logger.go:42: 09:39:14 | watcher-tls | skipping kubernetes event logging'
2026-03-14 09:44:27.418120 | controller |     - === CONT  kuttl/harness/watcher-rmquser
2026-03-14 09:44:27.418135 | controller |     - '    logger.go:42: 09:39:14 | watcher-rmquser | Skipping creation of user-supplied
2026-03-14 09:44:27.418142 | controller |       namespace: watcher-kuttl-default'
2026-03-14 09:44:27.418148 | controller |     - '    logger.go:42: 09:39:14 | watcher-rmquser/0-cleanup-watcher | starting test
2026-03-14 09:44:27.418153 | controller |       step 0-cleanup-watcher'
2026-03-14 09:44:27.418165 | controller |     - '    logger.go:42: 09:39:14 | watcher-rmquser/0-cleanup-watcher | test step completed
2026-03-14 09:44:27.418197 | controller |       0-cleanup-watcher'
2026-03-14 09:44:27.418203 | controller |     - '    logger.go:42: 09:39:14 | watcher-rmquser/1-deploy | starting test step 1-deploy'
2026-03-14 09:44:27.418210 | controller |     - '    logger.go:42: 09:39:14 | watcher-rmquser/1-deploy | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:44:27.418216 | controller |       created'
2026-03-14 09:44:27.418222 | controller |     - '    logger.go:42: 09:39:14 | watcher-rmquser/1-deploy | running command: [sh -c
2026-03-14 09:44:27.418228 | controller |       set -euxo pipefail'
2026-03-14 09:44:27.418234 | controller |     - '        '
2026-03-14 09:44:27.418239 | controller |     - '        # Wait for Watcher to be Ready'
2026-03-14 09:44:27.418245 | controller |     - '        kubectl wait --for=condition=Ready watcher/watcher-kuttl -n $NAMESPACE
2026-03-14 09:44:27.418251 | controller |       --timeout=300s'
2026-03-14 09:44:27.418257 | controller |     - '        '
2026-03-14 09:44:27.418263 | controller |     - '        # Verify WatcherNotificationTransportURLReady condition exists and is True'
2026-03-14 09:44:27.418272 | controller |     - '        kubectl get watcher watcher-kuttl -n $NAMESPACE -o jsonpath=''{.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}''
2026-03-14 09:44:27.418278 | controller |       | grep -q "True"'
2026-03-14 09:44:27.418284 | controller |     - '        echo "WatcherNotificationTransportURLReady condition is True"'
2026-03-14 09:44:27.418304 | controller |     - '        '
2026-03-14 09:44:27.418313 | controller |     - '        # Count TransportURL CRs - should be exactly 2 (one for messaging, one
2026-03-14 09:44:27.418319 | controller |       for notifications)'
2026-03-14 09:44:27.418325 | controller |     - '        transport_count=$(kubectl get transporturl -n $NAMESPACE -o name | grep
2026-03-14 09:44:27.418331 | controller |       "watcher-kuttl-watcher-transport" | wc -l)'
2026-03-14 09:44:27.418336 | controller |     - '        notification_transport_count=$(kubectl get transporturl -n $NAMESPACE -o
2026-03-14 09:44:27.418342 | controller |       name | grep "watcher-kuttl-watcher-notification" | wc -l)'
2026-03-14 09:44:27.418348 | controller |     - '        '
2026-03-14 09:44:27.418354 | controller |     - '        if [ "$transport_count" -ne "1" ]; then'
2026-03-14 09:44:27.418360 | controller |     - '          echo "Expected 1 watcher-transport TransportURL, found $transport_count"'
2026-03-14 09:44:27.418366 | controller |     - '          exit 1'
2026-03-14 09:44:27.418372 | controller |     - '        fi'
2026-03-14 09:44:27.418381 | controller |     - '        '
2026-03-14 09:44:27.418387 | controller |     - '        if [ "$notification_transport_count" -ne "1" ]; then'
2026-03-14 09:44:27.418393 | controller |     - '          echo "Expected 1 notification-transport TransportURL, found $notification_transport_count"'
2026-03-14 09:44:27.418398 | controller |     - '          exit 1'
2026-03-14 09:44:27.418404 | controller |     - '        fi'
2026-03-14 09:44:27.418410 | controller |     - '        '
2026-03-14 09:44:27.418416 | controller |     - '        echo "Correctly found 2 TransportURLs (separate clusters: transport and
2026-03-14 09:44:27.418422 | controller |       notification)"'
2026-03-14 09:44:27.418427 | controller |     - '        '
2026-03-14 09:44:27.418433 | controller |     - '        # Verify watcher-transport has correct user and vhost'
2026-03-14 09:44:27.418439 | controller |     - '        transport_user=$(kubectl get transporturl watcher-kuttl-watcher-transport
2026-03-14 09:44:27.418445 | controller |       -n $NAMESPACE -o jsonpath=''{.spec.username}'')'
2026-03-14 09:44:27.418451 | controller |     - '        transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport
2026-03-14 09:44:27.418457 | controller |       -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')'
2026-03-14 09:44:27.418463 | controller |     - '        if [ "$transport_user" != "watcher-rpc" ]; then'
2026-03-14 09:44:27.418469 | controller |     - '          echo "Expected watcher-transport username ''watcher-rpc'', found ''$transport_user''"'
2026-03-14 09:44:27.418474 | controller |     - '          exit 1'
2026-03-14 09:44:27.418480 | controller |     - '        fi'
2026-03-14 09:44:27.418486 | controller |     - '        if [ "$transport_vhost" != "watcher-rpc" ]; then'
2026-03-14 09:44:27.418492 | controller |     - '          echo "Expected watcher-transport vhost ''watcher-rpc'', found ''$transport_vhost''"'
2026-03-14 09:44:27.418497 | controller |     - '          exit 1'
2026-03-14 09:44:27.418503 | controller |     - '        fi'
2026-03-14 09:44:27.418509 | controller |     - '        echo "Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)"'
2026-03-14 09:44:27.418515 | controller |     - '        '
2026-03-14 09:44:27.418523 | controller |     - '        # Verify notification-transport has correct user and vhost'
2026-03-14 09:44:27.418529 | controller |     - '        notif_user=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications
2026-03-14 09:44:27.418535 | controller |       -n $NAMESPACE -o jsonpath=''{.spec.username}'')'
2026-03-14 09:44:27.418541 | controller |     - '        notif_vhost=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications
2026-03-14 09:44:27.418547 | controller |       -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')'
2026-03-14 09:44:27.418552 | controller |     - '        if [ "$notif_user" != "watcher-notifications" ]; then'
2026-03-14 09:44:27.418558 | controller |     - '          echo "Expected notification-transport username ''watcher-notifications'',
2026-03-14 09:44:27.418566 | controller |       found ''$notif_user''"'
2026-03-14 09:44:27.418572 | controller |     - '          exit 1'
2026-03-14 09:44:27.418578 | controller |     - '        fi'
2026-03-14 09:44:27.418584 | controller |     - '        if [ "$notif_vhost" != "watcher-notifications" ]; then'
2026-03-14 09:44:27.418590 | controller |     - '          echo "Expected notification-transport vhost ''watcher-notifications'',
2026-03-14 09:44:27.418596 | controller |       found ''$notif_vhost''"'
2026-03-14 09:44:27.418601 | controller |     - '          exit 1'
2026-03-14 09:44:27.418607 | controller |     - '        fi'
2026-03-14 09:44:27.418613 | controller |     - '        echo "Notification transport has correct user (watcher-notifications) and
2026-03-14 09:44:27.418630 | controller |       vhost (watcher-notifications)"'
2026-03-14 09:44:27.418637 | controller |     - '        '
2026-03-14 09:44:27.418644 | controller |     - '        # Verify that watcher.conf contains the notifications transport_url'
2026-03-14 09:44:27.418649 | controller |     - '        WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l "service=watcher-api"
2026-03-14 09:44:27.418655 | controller |       -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)'
2026-03-14 09:44:27.418661 | controller |     - '        if [ -z "${WATCHER_API_POD}" ]; then'
2026-03-14 09:44:27.418667 | controller |     - '          echo "No watcher-api pod found"'
2026-03-14 09:44:27.418673 | controller |     - '          exit 1'
2026-03-14 09:44:27.418678 | controller |     - '        fi'
2026-03-14 09:44:27.418684 | controller |     - '        # Verify RPC transport_url in DEFAULT section'
2026-03-14 09:44:27.418690 | controller |     - '        rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api
2026-03-14 09:44:27.418696 | controller |       -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E ''^\[DEFAULT\]'' -A
2026-03-14 09:44:27.418702 | controller |       50 | grep ''transport_url'' | head -1 || true)'
2026-03-14 09:44:27.418708 | controller |     - '        if [ -z "$rpc_transport_url" ]; then'
2026-03-14 09:44:27.418714 | controller |     - '          echo "transport_url not found in DEFAULT section"'
2026-03-14 09:44:27.418720 | controller |     - '          exit 1'
2026-03-14 09:44:27.418725 | controller |     - '        fi'
2026-03-14 09:44:27.418731 | controller |     - '        echo "Found RPC transport_url: $rpc_transport_url"'
2026-03-14 09:44:27.418737 | controller |     - '        '
2026-03-14 09:44:27.418743 | controller |     - '        # Verify the RPC transport_url contains the correct vhost (watcher-rpc)'
2026-03-14 09:44:27.418749 | controller |     - '        if ! echo "$rpc_transport_url" | grep -q ''/watcher-rpc''; then'
2026-03-14 09:44:27.418755 | controller |     - '          echo "RPC transport_url does not contain expected vhost ''/watcher-rpc''"'
2026-03-14 09:44:27.418760 | controller |     - '          exit 1'
2026-03-14 09:44:27.418766 | controller |     - '        fi'
2026-03-14 09:44:27.418772 | controller |     - '        echo "Successfully verified vhost ''watcher-rpc'' in RPC transport_url"'
2026-03-14 09:44:27.418778 | controller |     - '        '
2026-03-14 09:44:27.418784 | controller |     - '        # Verify the RPC transport_url contains the correct username (watcher-rpc)'
2026-03-14 09:44:27.418790 | controller |     - '        if ! echo "$rpc_transport_url" | grep -q ''watcher-rpc:''; then'
2026-03-14 09:44:27.418796 | controller |     - '          echo "RPC transport_url does not contain expected username ''watcher-rpc:''"'
2026-03-14 09:44:27.418802 | controller |     - '          exit 1'
2026-03-14 09:44:27.418807 | controller |     - '        fi'
2026-03-14 09:44:27.418813 | controller |     - '        echo "Successfully verified username ''watcher-rpc'' in RPC transport_url"'
2026-03-14 09:44:27.418819 | controller |     - '        '
2026-03-14 09:44:27.418825 | controller |     - '        # Verify oslo_messaging_notifications section has transport_url configured'
2026-03-14 09:44:27.418831 | controller |     - '        notif_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c
2026-03-14 09:44:27.418837 | controller |       watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -A 5 ''\[oslo_messaging_notifications\]''
2026-03-14 09:44:27.418843 | controller |       | grep ''transport_url'' || true)'
2026-03-14 09:44:27.418848 | controller |     - '        if [ -z "$notif_transport_url" ]; then'
2026-03-14 09:44:27.418854 | controller |     - '          echo "transport_url not found in oslo_messaging_notifications section"'
2026-03-14 09:44:27.418863 | controller |     - '          exit 1'
2026-03-14 09:44:27.418869 | controller |     - '        fi'
2026-03-14 09:44:27.418875 | controller |     - '        echo "Found notifications transport_url: $notif_transport_url"'
2026-03-14 09:44:27.418880 | controller |     - '        '
2026-03-14 09:44:27.418886 | controller |     - '        # Verify the notifications transport_url contains the correct vhost (watcher-notifications)'
2026-03-14 09:44:27.418895 | controller |     - '        if ! echo "$notif_transport_url" | grep -q ''/watcher-notifications'';
2026-03-14 09:44:27.418902 | controller |       then'
2026-03-14 09:44:27.418908 | controller |     - '          echo "Notifications transport_url does not contain expected vhost ''/watcher-notifications''"'
2026-03-14 09:44:27.418914 | controller |     - '          exit 1'
2026-03-14 09:44:27.418919 | controller |     - '        fi'
2026-03-14 09:44:27.418925 | controller |     - '        echo "Successfully verified vhost ''watcher-notifications'' in notifications
2026-03-14 09:44:27.418931 | controller |       transport_url"'
2026-03-14 09:44:27.418937 | controller |     - '        '
2026-03-14 09:44:27.418942 | controller |     - '        # Verify the notifications transport_url contains the correct username
2026-03-14 09:44:27.418948 | controller |       (watcher-notifications)'
2026-03-14 09:44:27.418954 | controller |     - '        if ! echo "$notif_transport_url" | grep -q ''watcher-notifications:'';
2026-03-14 09:44:27.418960 | controller |       then'
2026-03-14 09:44:27.418966 | controller |     - '          echo "Notifications transport_url does not contain expected username
2026-03-14 09:44:27.418971 | controller |       ''watcher-notifications:''"'
2026-03-14 09:44:27.418977 | controller |     - '          exit 1'
2026-03-14 09:44:27.418983 | controller |     - '        fi'
2026-03-14 09:44:27.418988 | controller |     - '        echo "Successfully verified username ''watcher-notifications'' in notifications
2026-03-14 09:44:27.418994 | controller |       transport_url"'
2026-03-14 09:44:27.418999 | controller |     - '        '
2026-03-14 09:44:27.419003 | controller |     - '        exit 0'
2026-03-14 09:44:27.419008 | controller |     - '        ]'
2026-03-14 09:44:27.419012 | controller |     - '    logger.go:42: 09:39:14 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready
2026-03-14 09:44:27.419016 | controller |       watcher/watcher-kuttl -n watcher-kuttl-default --timeout=300s'
2026-03-14 09:44:27.419020 | controller |     - '    logger.go:42: 09:39:39 | watcher-rmquser/1-deploy | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:44:27.419026 | controller |       condition met'
2026-03-14 09:44:27.419032 | controller |     - '    logger.go:42: 09:39:39 | watcher-rmquser/1-deploy | + kubectl get watcher watcher-kuttl
2026-03-14 09:44:27.419038 | controller |       -n watcher-kuttl-default -o ''jsonpath={.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}'''
2026-03-14 09:44:27.419044 | controller |     - '    logger.go:42: 09:39:39 | watcher-rmquser/1-deploy | + grep -q True'
2026-03-14 09:44:27.419058 | controller |     - '    logger.go:42: 09:39:39 | watcher-rmquser/1-deploy | + echo ''WatcherNotificationTransportURLReady
2026-03-14 09:44:27.419065 | controller |       condition is True'''
2026-03-14 09:44:27.419071 | controller |     - '    logger.go:42: 09:39:39 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady
2026-03-14 09:44:27.419077 | controller |       condition is True'
2026-03-14 09:44:27.419083 | controller |     - '    logger.go:42: 09:39:39 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:44:27.419088 | controller |       -n watcher-kuttl-default -o name'
2026-03-14 09:44:27.419094 | controller |     - '    logger.go:42: 09:39:39 | watcher-rmquser/1-deploy | ++ wc -l'
2026-03-14 09:44:27.419100 | controller |     - '    logger.go:42: 09:39:39 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-transport'
2026-03-14 09:44:27.419106 | controller |     - '    logger.go:42: 09:39:39 | watcher-rmquser/1-deploy | + transport_count=1'
2026-03-14 09:44:27.419112 | controller |     - '    logger.go:42: 09:39:39 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:44:27.419117 | controller |       -n watcher-kuttl-default -o name'
2026-03-14 09:44:27.419123 | controller |     - '    logger.go:42: 09:39:39 | watcher-rmquser/1-deploy | ++ wc -l'
2026-03-14 09:44:27.419129 | controller |     - '    logger.go:42: 09:39:39 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-notification'
2026-03-14 09:44:27.419140 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + notification_transport_count=1'
2026-03-14 09:44:27.419145 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']'''
2026-03-14 09:44:27.419151 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']'''
2026-03-14 09:44:27.419157 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + echo ''Correctly found
2026-03-14 09:44:27.419163 | controller |       2 TransportURLs (separate clusters: transport and notification)'''
2026-03-14 09:44:27.419168 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs
2026-03-14 09:44:27.419174 | controller |       (separate clusters: transport and notification)'
2026-03-14 09:44:27.419180 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:44:27.419186 | controller |       watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.username}'''
2026-03-14 09:44:27.419192 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + transport_user=watcher-rpc'
2026-03-14 09:44:27.419197 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:44:27.419203 | controller |       watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.vhost}'''
2026-03-14 09:44:27.419209 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + transport_vhost=watcher-rpc'
2026-03-14 09:44:27.419215 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!=''
2026-03-14 09:44:27.419221 | controller |       watcher-rpc '']'''
2026-03-14 09:44:27.419226 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!=''
2026-03-14 09:44:27.419230 | controller |       watcher-rpc '']'''
2026-03-14 09:44:27.419235 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + echo ''Watcher transport
2026-03-14 09:44:27.419239 | controller |       has correct user (watcher-rpc) and vhost (watcher-rpc)'''
2026-03-14 09:44:27.419243 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | Watcher transport has correct
2026-03-14 09:44:27.419247 | controller |       user (watcher-rpc) and vhost (watcher-rpc)'
2026-03-14 09:44:27.419251 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:44:27.419256 | controller |       watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default
2026-03-14 09:44:27.419262 | controller |       -o ''jsonpath={.spec.username}'''
2026-03-14 09:44:27.419268 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + notif_user=watcher-notifications'
2026-03-14 09:44:27.419274 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:44:27.419279 | controller |       watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default
2026-03-14 09:44:27.419285 | controller |       -o ''jsonpath={.spec.vhost}'''
2026-03-14 09:44:27.419311 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications'
2026-03-14 09:44:27.419316 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + ''['' watcher-notifications
2026-03-14 09:44:27.419320 | controller |       ''!='' watcher-notifications '']'''
2026-03-14 09:44:27.419327 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + ''['' watcher-notifications
2026-03-14 09:44:27.419332 | controller |       ''!='' watcher-notifications '']'''
2026-03-14 09:44:27.419336 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + echo ''Notification transport
2026-03-14 09:44:27.419340 | controller |       has correct user (watcher-notifications) and vhost (watcher-notifications)'''
2026-03-14 09:44:27.419345 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | Notification transport
2026-03-14 09:44:27.419350 | controller |       has correct user (watcher-notifications) and vhost (watcher-notifications)'
2026-03-14 09:44:27.419355 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ kubectl get pods -n
2026-03-14 09:44:27.419361 | controller |       watcher-kuttl-default -l service=watcher-api -o custom-columns=:metadata.name --no-headers'
2026-03-14 09:44:27.419373 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ head -1'
2026-03-14 09:44:27.419378 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ grep -v ''^$'''
2026-03-14 09:44:27.419382 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + WATCHER_API_POD=watcher-kuttl-api-0'
2026-03-14 09:44:27.419386 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + ''['' -z watcher-kuttl-api-0
2026-03-14 09:44:27.419391 | controller |       '']'''
2026-03-14 09:44:27.419395 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default
2026-03-14 09:44:27.419399 | controller |       watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.419403 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ grep transport_url'
2026-03-14 09:44:27.419407 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ grep -E ''^\[DEFAULT\]''
2026-03-14 09:44:27.419412 | controller |       -A 50'
2026-03-14 09:44:27.419416 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ head -1'
2026-03-14 09:44:27.419420 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + rpc_transport_url=''transport_url
2026-03-14 09:44:27.419424 | controller |       = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'''
2026-03-14 09:44:27.419429 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url
2026-03-14 09:44:27.419433 | controller |       = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''
2026-03-14 09:44:27.419437 | controller |       '']'''
2026-03-14 09:44:27.419441 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + echo ''Found RPC transport_url:
2026-03-14 09:44:27.419446 | controller |       transport_url = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'''
2026-03-14 09:44:27.419450 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | Found RPC transport_url:
2026-03-14 09:44:27.419454 | controller |       transport_url = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'
2026-03-14 09:44:27.419459 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + echo ''transport_url
2026-03-14 09:44:27.419463 | controller |       = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'''
2026-03-14 09:44:27.419467 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc'
2026-03-14 09:44:27.419471 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + echo ''Successfully verified
2026-03-14 09:44:27.419476 | controller |       vhost ''\''''watcher-rpc''\'''' in RPC transport_url'''
2026-03-14 09:44:27.419480 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | Successfully verified vhost
2026-03-14 09:44:27.419484 | controller |       ''watcher-rpc'' in RPC transport_url'
2026-03-14 09:44:27.419488 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + echo ''transport_url
2026-03-14 09:44:27.419492 | controller |       = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'''
2026-03-14 09:44:27.419497 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:'
2026-03-14 09:44:27.419501 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | + echo ''Successfully verified
2026-03-14 09:44:27.419505 | controller |       username ''\''''watcher-rpc''\'''' in RPC transport_url'''
2026-03-14 09:44:27.419509 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | Successfully verified username
2026-03-14 09:44:27.419513 | controller |       ''watcher-rpc'' in RPC transport_url'
2026-03-14 09:44:27.419518 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ grep -A 5 ''\[oslo_messaging_notifications\]'''
2026-03-14 09:44:27.419524 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ grep transport_url'
2026-03-14 09:44:27.419530 | controller |     - '    logger.go:42: 09:39:40 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default
2026-03-14 09:44:27.419534 | controller |       watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.419538 | controller |     - '    logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | + notif_transport_url=''transport_url
2026-03-14 09:44:27.419542 | controller |       = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'''
2026-03-14 09:44:27.419547 | controller |     - '    logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url
2026-03-14 09:44:27.419551 | controller |       = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''
2026-03-14 09:44:27.419555 | controller |       '']'''
2026-03-14 09:44:27.419559 | controller |     - '    logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | + echo ''Found notifications
2026-03-14 09:44:27.419564 | controller |       transport_url: transport_url = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'''
2026-03-14 09:44:27.419574 | controller |     - '    logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | Found notifications transport_url:
2026-03-14 09:44:27.419579 | controller |       transport_url = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'
2026-03-14 09:44:27.419583 | controller |     - '    logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | + echo ''transport_url
2026-03-14 09:44:27.419587 | controller |       = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'''
2026-03-14 09:44:27.419592 | controller |     - '    logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications'
2026-03-14 09:44:27.419596 | controller |     - '    logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | + echo ''Successfully verified
2026-03-14 09:44:27.419600 | controller |       vhost ''\''''watcher-notifications''\'''' in notifications transport_url'''
2026-03-14 09:44:27.419604 | controller |     - '    logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | Successfully verified vhost
2026-03-14 09:44:27.419608 | controller |       ''watcher-notifications'' in notifications transport_url'
2026-03-14 09:44:27.419613 | controller |     - '    logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | + grep -q watcher-notifications:'
2026-03-14 09:44:27.419617 | controller |     - '    logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | + echo ''transport_url
2026-03-14 09:44:27.419621 | controller |       = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'''
2026-03-14 09:44:27.419625 | controller |     - '    logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | + echo ''Successfully verified
2026-03-14 09:44:27.419630 | controller |       username ''\''''watcher-notifications''\'''' in notifications transport_url'''
2026-03-14 09:44:27.419634 | controller |     - '    logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | Successfully verified username
2026-03-14 09:44:27.419638 | controller |       ''watcher-notifications'' in notifications transport_url'
2026-03-14 09:44:27.419642 | controller |     - '    logger.go:42: 09:39:41 | watcher-rmquser/1-deploy | + exit 0'
2026-03-14 09:44:27.419646 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | running command: [sh -c
2026-03-14 09:44:27.419652 | controller |       set -euxo pipefail'
2026-03-14 09:44:27.419658 | controller |     - '        '
2026-03-14 09:44:27.419664 | controller |     - '        # Wait for Watcher to be Ready'
2026-03-14 09:44:27.419670 | controller |     - '        kubectl wait --for=condition=Ready watcher/watcher-kuttl -n $NAMESPACE
2026-03-14 09:44:27.419676 | controller |       --timeout=300s'
2026-03-14 09:44:27.419778 | controller |     - '        '
2026-03-14 09:44:27.419792 | controller |     - '        # Verify WatcherNotificationTransportURLReady condition exists and is True'
2026-03-14 09:44:27.419796 | controller |     - '        kubectl get watcher watcher-kuttl -n $NAMESPACE -o jsonpath=''{.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}''
2026-03-14 09:44:27.419802 | controller |       | grep -q "True"'
2026-03-14 09:44:27.419806 | controller |     - '        echo "WatcherNotificationTransportURLReady condition is True"'
2026-03-14 09:44:27.419812 | controller |     - '        '
2026-03-14 09:44:27.419817 | controller |     - '        # Count TransportURL CRs - should be exactly 2 (one for messaging, one
2026-03-14 09:44:27.419821 | controller |       for notifications)'
2026-03-14 09:44:27.419826 | controller |     - '        transport_count=$(kubectl get transporturl -n $NAMESPACE -o name | grep
2026-03-14 09:44:27.419830 | controller |       "watcher-kuttl-watcher-transport" | wc -l)'
2026-03-14 09:44:27.419834 | controller |     - '        notification_transport_count=$(kubectl get transporturl -n $NAMESPACE -o
2026-03-14 09:44:27.419839 | controller |       name | grep "watcher-kuttl-watcher-notification" | wc -l)'
2026-03-14 09:44:27.419843 | controller |     - '        '
2026-03-14 09:44:27.419847 | controller |     - '        if [ "$transport_count" -ne "1" ]; then'
2026-03-14 09:44:27.419851 | controller |     - '          echo "Expected 1 watcher-transport TransportURL, found $transport_count"'
2026-03-14 09:44:27.419856 | controller |     - '          exit 1'
2026-03-14 09:44:27.419860 | controller |     - '        fi'
2026-03-14 09:44:27.419864 | controller |     - '        '
2026-03-14 09:44:27.419869 | controller |     - '        if [ "$notification_transport_count" -ne "1" ]; then'
2026-03-14 09:44:27.419873 | controller |     - '          echo "Expected 1 notification-transport TransportURL, found $notification_transport_count"'
2026-03-14 09:44:27.419877 | controller |     - '          exit 1'
2026-03-14 09:44:27.419882 | controller |     - '        fi'
2026-03-14 09:44:27.419886 | controller |     - '        '
2026-03-14 09:44:27.419890 | controller |     - '        echo "Correctly found 2 TransportURLs (separate clusters: transport and
2026-03-14 09:44:27.419895 | controller |       notification)"'
2026-03-14 09:44:27.419901 | controller |     - '        '
2026-03-14 09:44:27.419905 | controller |     - '        # Verify watcher-transport has correct user and vhost'
2026-03-14 09:44:27.419909 | controller |     - '        transport_user=$(kubectl get transporturl watcher-kuttl-watcher-transport
2026-03-14 09:44:27.419914 | controller |       -n $NAMESPACE -o jsonpath=''{.spec.username}'')'
2026-03-14 09:44:27.419919 | controller |     - '        transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport
2026-03-14 09:44:27.419923 | controller |       -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')'
2026-03-14 09:44:27.419927 | controller |     - '        if [ "$transport_user" != "watcher-rpc" ]; then'
2026-03-14 09:44:27.419932 | controller |     - '          echo "Expected watcher-transport username ''watcher-rpc'', found ''$transport_user''"'
2026-03-14 09:44:27.419936 | controller |     - '          exit 1'
2026-03-14 09:44:27.419940 | controller |     - '        fi'
2026-03-14 09:44:27.419944 | controller |     - '        if [ "$transport_vhost" != "watcher-rpc" ]; then'
2026-03-14 09:44:27.419949 | controller |     - '          echo "Expected watcher-transport vhost ''watcher-rpc'', found ''$transport_vhost''"'
2026-03-14 09:44:27.419953 | controller |     - '          exit 1'
2026-03-14 09:44:27.419964 | controller |     - '        fi'
2026-03-14 09:44:27.419969 | controller |     - '        echo "Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)"'
2026-03-14 09:44:27.419973 | controller |     - '        '
2026-03-14 09:44:27.419978 | controller |     - '        # Verify notification-transport has correct user and vhost'
2026-03-14 09:44:27.419982 | controller |     - '        notif_user=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications
2026-03-14 09:44:27.419986 | controller |       -n $NAMESPACE -o jsonpath=''{.spec.username}'')'
2026-03-14 09:44:27.419991 | controller |     - '        notif_vhost=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications
2026-03-14 09:44:27.419995 | controller |       -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')'
2026-03-14 09:44:27.419999 | controller |     - '        if [ "$notif_user" != "watcher-notifications" ]; then'
2026-03-14 09:44:27.420005 | controller |     - '          echo "Expected notification-transport username ''watcher-notifications'',
2026-03-14 09:44:27.420010 | controller |       found ''$notif_user''"'
2026-03-14 09:44:27.420014 | controller |     - '          exit 1'
2026-03-14 09:44:27.420019 | controller |     - '        fi'
2026-03-14 09:44:27.420023 | controller |     - '        if [ "$notif_vhost" != "watcher-notifications" ]; then'
2026-03-14 09:44:27.420028 | controller |     - '          echo "Expected notification-transport vhost ''watcher-notifications'',
2026-03-14 09:44:27.420032 | controller |       found ''$notif_vhost''"'
2026-03-14 09:44:27.420036 | controller |     - '          exit 1'
2026-03-14 09:44:27.420041 | controller |     - '        fi'
2026-03-14 09:44:27.420045 | controller |     - '        echo "Notification transport has correct user (watcher-notifications) and
2026-03-14 09:44:27.420049 | controller |       vhost (watcher-notifications)"'
2026-03-14 09:44:27.420053 | controller |     - '        '
2026-03-14 09:44:27.420058 | controller |     - '        # Verify that watcher.conf contains the notifications transport_url'
2026-03-14 09:44:27.420062 | controller |     - '        WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l "service=watcher-api"
2026-03-14 09:44:27.420066 | controller |       -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)'
2026-03-14 09:44:27.420071 | controller |     - '        if [ -z "${WATCHER_API_POD}" ]; then'
2026-03-14 09:44:27.420075 | controller |     - '          echo "No watcher-api pod found"'
2026-03-14 09:44:27.420079 | controller |     - '          exit 1'
2026-03-14 09:44:27.420084 | controller |     - '        fi'
2026-03-14 09:44:27.420088 | controller |     - '        # Verify RPC transport_url in DEFAULT section'
2026-03-14 09:44:27.420092 | controller |     - '        rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api
2026-03-14 09:44:27.420097 | controller |       -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E ''^\[DEFAULT\]'' -A
2026-03-14 09:44:27.420101 | controller |       50 | grep ''transport_url'' | head -1 || true)'
2026-03-14 09:44:27.420105 | controller |     - '        if [ -z "$rpc_transport_url" ]; then'
2026-03-14 09:44:27.420109 | controller |     - '          echo "transport_url not found in DEFAULT section"'
2026-03-14 09:44:27.420114 | controller |     - '          exit 1'
2026-03-14 09:44:27.420118 | controller |     - '        fi'
2026-03-14 09:44:27.420122 | controller |     - '        echo "Found RPC transport_url: $rpc_transport_url"'
2026-03-14 09:44:27.420127 | controller |     - '        '
2026-03-14 09:44:27.420131 | controller |     - '        # Verify the RPC transport_url contains the correct vhost (watcher-rpc)'
2026-03-14 09:44:27.420135 | controller |     - '        if ! echo "$rpc_transport_url" | grep -q ''/watcher-rpc''; then'
2026-03-14 09:44:27.420139 | controller |     - '          echo "RPC transport_url does not contain expected vhost ''/watcher-rpc''"'
2026-03-14 09:44:27.420144 | controller |     - '          exit 1'
2026-03-14 09:44:27.420148 | controller |     - '        fi'
2026-03-14 09:44:27.420152 | controller |     - '        echo "Successfully verified vhost ''watcher-rpc'' in RPC transport_url"'
2026-03-14 09:44:27.420156 | controller |     - '        '
2026-03-14 09:44:27.420161 | controller |     - '        # Verify the RPC transport_url contains the correct username (watcher-rpc)'
2026-03-14 09:44:27.420165 | controller |     - '        if ! echo "$rpc_transport_url" | grep -q ''watcher-rpc:''; then'
2026-03-14 09:44:27.420169 | controller |     - '          echo "RPC transport_url does not contain expected username ''watcher-rpc:''"'
2026-03-14 09:44:27.420173 | controller |     - '          exit 1'
2026-03-14 09:44:27.420178 | controller |     - '        fi'
2026-03-14 09:44:27.420182 | controller |     - '        echo "Successfully verified username ''watcher-rpc'' in RPC transport_url"'
2026-03-14 09:44:27.420186 | controller |     - '        '
2026-03-14 09:44:27.420190 | controller |     - '        # Verify oslo_messaging_notifications section has transport_url configured'
2026-03-14 09:44:27.420195 | controller |     - '        notif_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c
2026-03-14 09:44:27.420199 | controller |       watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -A 5 ''\[oslo_messaging_notifications\]''
2026-03-14 09:44:27.420203 | controller |       | grep ''transport_url'' || true)'
2026-03-14 09:44:27.420207 | controller |     - '        if [ -z "$notif_transport_url" ]; then'
2026-03-14 09:44:27.420214 | controller |     - '          echo "transport_url not found in oslo_messaging_notifications section"'
2026-03-14 09:44:27.420218 | controller |     - '          exit 1'
2026-03-14 09:44:27.420222 | controller |     - '        fi'
2026-03-14 09:44:27.420227 | controller |     - '        echo "Found notifications transport_url: $notif_transport_url"'
2026-03-14 09:44:27.420231 | controller |     - '        '
2026-03-14 09:44:27.420235 | controller |     - '        # Verify the notifications transport_url contains the correct vhost (watcher-notifications)'
2026-03-14 09:44:27.420239 | controller |     - '        if ! echo "$notif_transport_url" | grep -q ''/watcher-notifications'';
2026-03-14 09:44:27.420244 | controller |       then'
2026-03-14 09:44:27.420248 | controller |     - '          echo "Notifications transport_url does not contain expected vhost ''/watcher-notifications''"'
2026-03-14 09:44:27.420252 | controller |     - '          exit 1'
2026-03-14 09:44:27.420256 | controller |     - '        fi'
2026-03-14 09:44:27.420261 | controller |     - '        echo "Successfully verified vhost ''watcher-notifications'' in notifications
2026-03-14 09:44:27.420265 | controller |       transport_url"'
2026-03-14 09:44:27.420269 | controller |     - '        '
2026-03-14 09:44:27.420279 | controller |     - '        # Verify the notifications transport_url contains the correct username
2026-03-14 09:44:27.420284 | controller |       (watcher-notifications)'
2026-03-14 09:44:27.420303 | controller |     - '        if ! echo "$notif_transport_url" | grep -q ''watcher-notifications:'';
2026-03-14 09:44:27.420311 | controller |       then'
2026-03-14 09:44:27.420316 | controller |     - '          echo "Notifications transport_url does not contain expected username
2026-03-14 09:44:27.420320 | controller |       ''watcher-notifications:''"'
2026-03-14 09:44:27.420325 | controller |     - '          exit 1'
2026-03-14 09:44:27.420329 | controller |     - '        fi'
2026-03-14 09:44:27.420333 | controller |     - '        echo "Successfully verified username ''watcher-notifications'' in notifications
2026-03-14 09:44:27.420338 | controller |       transport_url"'
2026-03-14 09:44:27.420342 | controller |     - '        '
2026-03-14 09:44:27.420346 | controller |     - '        exit 0'
2026-03-14 09:44:27.420350 | controller |     - '        ]'
2026-03-14 09:44:27.420355 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready
2026-03-14 09:44:27.420359 | controller |       watcher/watcher-kuttl -n watcher-kuttl-default --timeout=300s'
2026-03-14 09:44:27.420363 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:44:27.420369 | controller |       condition met'
2026-03-14 09:44:27.420376 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | + kubectl get watcher watcher-kuttl
2026-03-14 09:44:27.420382 | controller |       -n watcher-kuttl-default -o ''jsonpath={.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}'''
2026-03-14 09:44:27.420388 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | + grep -q True'
2026-03-14 09:44:27.420394 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | + echo ''WatcherNotificationTransportURLReady
2026-03-14 09:44:27.420400 | controller |       condition is True'''
2026-03-14 09:44:27.420407 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady
2026-03-14 09:44:27.420413 | controller |       condition is True'
2026-03-14 09:44:27.420421 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:44:27.420425 | controller |       -n watcher-kuttl-default -o name'
2026-03-14 09:44:27.420429 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | ++ wc -l'
2026-03-14 09:44:27.420434 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-transport'
2026-03-14 09:44:27.420438 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | + transport_count=1'
2026-03-14 09:44:27.420442 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:44:27.420446 | controller |       -n watcher-kuttl-default -o name'
2026-03-14 09:44:27.420451 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | ++ wc -l'
2026-03-14 09:44:27.420455 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-notification'
2026-03-14 09:44:27.420461 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | + notification_transport_count=1'
2026-03-14 09:44:27.420466 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']'''
2026-03-14 09:44:27.420470 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']'''
2026-03-14 09:44:27.420474 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | + echo ''Correctly found
2026-03-14 09:44:27.420480 | controller |       2 TransportURLs (separate clusters: transport and notification)'''
2026-03-14 09:44:27.420484 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs
2026-03-14 09:44:27.420488 | controller |       (separate clusters: transport and notification)'
2026-03-14 09:44:27.420492 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:44:27.420497 | controller |       watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.username}'''
2026-03-14 09:44:27.420501 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | + transport_user=watcher-rpc'
2026-03-14 09:44:27.420505 | controller |     - '    logger.go:42: 09:39:42 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:44:27.420509 | controller |       watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.vhost}'''
2026-03-14 09:44:27.420513 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + transport_vhost=watcher-rpc'
2026-03-14 09:44:27.420517 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!=''
2026-03-14 09:44:27.420522 | controller |       watcher-rpc '']'''
2026-03-14 09:44:27.420526 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!=''
2026-03-14 09:44:27.420530 | controller |       watcher-rpc '']'''
2026-03-14 09:44:27.420534 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo ''Watcher transport
2026-03-14 09:44:27.420538 | controller |       has correct user (watcher-rpc) and vhost (watcher-rpc)'''
2026-03-14 09:44:27.420543 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | Watcher transport has correct
2026-03-14 09:44:27.420547 | controller |       user (watcher-rpc) and vhost (watcher-rpc)'
2026-03-14 09:44:27.420551 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:44:27.420555 | controller |       watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default
2026-03-14 09:44:27.420559 | controller |       -o ''jsonpath={.spec.username}'''
2026-03-14 09:44:27.420564 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + notif_user=watcher-notifications'
2026-03-14 09:44:27.420576 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | ++ kubectl get transporturl
2026-03-14 09:44:27.420580 | controller |       watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default
2026-03-14 09:44:27.420585 | controller |       -o ''jsonpath={.spec.vhost}'''
2026-03-14 09:44:27.420589 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications'
2026-03-14 09:44:27.420593 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + ''['' watcher-notifications
2026-03-14 09:44:27.420597 | controller |       ''!='' watcher-notifications '']'''
2026-03-14 09:44:27.420602 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + ''['' watcher-notifications
2026-03-14 09:44:27.420606 | controller |       ''!='' watcher-notifications '']'''
2026-03-14 09:44:27.420610 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo ''Notification transport
2026-03-14 09:44:27.420614 | controller |       has correct user (watcher-notifications) and vhost (watcher-notifications)'''
2026-03-14 09:44:27.420619 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | Notification transport
2026-03-14 09:44:27.420659 | controller |       has correct user (watcher-notifications) and vhost (watcher-notifications)'
2026-03-14 09:44:27.420667 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | ++ kubectl get pods -n
2026-03-14 09:44:27.420672 | controller |       watcher-kuttl-default -l service=watcher-api -o custom-columns=:metadata.name --no-headers'
2026-03-14 09:44:27.420676 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | ++ head -1'
2026-03-14 09:44:27.420680 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | ++ grep -v ''^$'''
2026-03-14 09:44:27.420684 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + WATCHER_API_POD=watcher-kuttl-api-0'
2026-03-14 09:44:27.420689 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + ''['' -z watcher-kuttl-api-0
2026-03-14 09:44:27.420693 | controller |       '']'''
2026-03-14 09:44:27.420697 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | ++ grep transport_url'
2026-03-14 09:44:27.420701 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | ++ grep -E ''^\[DEFAULT\]''
2026-03-14 09:44:27.420706 | controller |       -A 50'
2026-03-14 09:44:27.420710 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | ++ head -1'
2026-03-14 09:44:27.420714 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default
2026-03-14 09:44:27.420720 | controller |       watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.420724 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + rpc_transport_url=''transport_url
2026-03-14 09:44:27.420729 | controller |       = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'''
2026-03-14 09:44:27.420733 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url
2026-03-14 09:44:27.420737 | controller |       = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''
2026-03-14 09:44:27.420741 | controller |       '']'''
2026-03-14 09:44:27.420745 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo ''Found RPC transport_url:
2026-03-14 09:44:27.420750 | controller |       transport_url = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'''
2026-03-14 09:44:27.420754 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | Found RPC transport_url:
2026-03-14 09:44:27.420758 | controller |       transport_url = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'
2026-03-14 09:44:27.420762 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo ''transport_url
2026-03-14 09:44:27.420767 | controller |       = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'''
2026-03-14 09:44:27.420771 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc'
2026-03-14 09:44:27.420775 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo ''Successfully verified
2026-03-14 09:44:27.420779 | controller |       vhost ''\''''watcher-rpc''\'''' in RPC transport_url'''
2026-03-14 09:44:27.420783 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | Successfully verified vhost
2026-03-14 09:44:27.420787 | controller |       ''watcher-rpc'' in RPC transport_url'
2026-03-14 09:44:27.420792 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo ''transport_url
2026-03-14 09:44:27.420796 | controller |       = rabbit://watcher-rpc:IYCC0sBp5xUOAIQah0PrLRnfgo0oHHOU@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'''
2026-03-14 09:44:27.420800 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:'
2026-03-14 09:44:27.420804 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo ''Successfully verified
2026-03-14 09:44:27.420809 | controller |       username ''\''''watcher-rpc''\'''' in RPC transport_url'''
2026-03-14 09:44:27.420813 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | Successfully verified username
2026-03-14 09:44:27.420817 | controller |       ''watcher-rpc'' in RPC transport_url'
2026-03-14 09:44:27.420824 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default
2026-03-14 09:44:27.420836 | controller |       watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.420840 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | ++ grep transport_url'
2026-03-14 09:44:27.420844 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | ++ grep -A 5 ''\[oslo_messaging_notifications\]'''
2026-03-14 09:44:27.420849 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + notif_transport_url=''transport_url
2026-03-14 09:44:27.420853 | controller |       = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'''
2026-03-14 09:44:27.420857 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url
2026-03-14 09:44:27.420861 | controller |       = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''
2026-03-14 09:44:27.420865 | controller |       '']'''
2026-03-14 09:44:27.420870 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo ''Found notifications
2026-03-14 09:44:27.420874 | controller |       transport_url: transport_url = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'''
2026-03-14 09:44:27.420878 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | Found notifications transport_url:
2026-03-14 09:44:27.420882 | controller |       transport_url = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'
2026-03-14 09:44:27.420887 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo ''transport_url
2026-03-14 09:44:27.420891 | controller |       = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'''
2026-03-14 09:44:27.420895 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications'
2026-03-14 09:44:27.420899 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo ''Successfully verified
2026-03-14 09:44:27.420903 | controller |       vhost ''\''''watcher-notifications''\'''' in notifications transport_url'''
2026-03-14 09:44:27.420908 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | Successfully verified vhost
2026-03-14 09:44:27.420912 | controller |       ''watcher-notifications'' in notifications transport_url'
2026-03-14 09:44:27.420916 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + grep -q watcher-notifications:'
2026-03-14 09:44:27.420921 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo ''transport_url
2026-03-14 09:44:27.420926 | controller |       = rabbit://watcher-notifications:UT9KJEaLOwx3SsYLMnKeiCAdRw5cC1-7@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'''
2026-03-14 09:44:27.420931 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + echo ''Successfully verified
2026-03-14 09:44:27.420935 | controller |       username ''\''''watcher-notifications''\'''' in notifications transport_url'''
2026-03-14 09:44:27.420941 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | Successfully verified username
2026-03-14 09:44:27.420946 | controller |       ''watcher-notifications'' in notifications transport_url'
2026-03-14 09:44:27.420950 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | + exit 0'
2026-03-14 09:44:27.420955 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/1-deploy | test step completed 1-deploy'
2026-03-14 09:44:27.420959 | controller |     - '    logger.go:42: 09:39:43 | watcher-rmquser/2-cleanup-watcher | starting test
2026-03-14 09:44:27.420963 | controller |       step 2-cleanup-watcher'
2026-03-14 09:44:27.420967 | controller |     - '    logger.go:42: 09:39:49 | watcher-rmquser/2-cleanup-watcher | test step completed
2026-03-14 09:44:27.420973 | controller |       2-cleanup-watcher'
2026-03-14 09:44:27.420977 | controller |     - '    logger.go:42: 09:39:49 | watcher-rmquser | skipping kubernetes event logging'
2026-03-14 09:44:27.420981 | controller |     - === CONT  kuttl/harness/watcher
2026-03-14 09:44:27.420986 | controller |     - '    logger.go:42: 09:39:49 | watcher | Skipping creation of user-supplied namespace:
2026-03-14 09:44:27.420990 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.420994 | controller |     - '    logger.go:42: 09:39:49 | watcher/0-cleanup-watcher | starting test step 0-cleanup-watcher'
2026-03-14 09:44:27.420998 | controller |     - '    logger.go:42: 09:39:49 | watcher/0-cleanup-watcher | test step completed 0-cleanup-watcher'
2026-03-14 09:44:27.421002 | controller |     - '    logger.go:42: 09:39:49 | watcher/1-deploy-with-defaults | starting test step
2026-03-14 09:44:27.421007 | controller |       1-deploy-with-defaults'
2026-03-14 09:44:27.421011 | controller |     - '    logger.go:42: 09:39:49 | watcher/1-deploy-with-defaults | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:44:27.421015 | controller |       created'
2026-03-14 09:44:27.421019 | controller |     - '    logger.go:42: 09:39:49 | watcher/1-deploy-with-defaults | running command:
2026-03-14 09:44:27.421024 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.421028 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.421032 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.421044 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:44:27.421049 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.421053 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.421057 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.421061 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.421066 | controller |       ]'
2026-03-14 09:44:27.421070 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.421074 | controller |       -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:44:27.421078 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.421082 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')"
2026-03-14 09:44:27.421087 | controller |       == 2 ]'
2026-03-14 09:44:27.421091 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.421095 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')"
2026-03-14 09:44:27.421099 | controller |       -ge 1 ]'
2026-03-14 09:44:27.421106 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.421110 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.421115 | controller |       == "" ]; then'
2026-03-14 09:44:27.421119 | controller |     - '            exit 0'
2026-03-14 09:44:27.421123 | controller |     - '        fi'
2026-03-14 09:44:27.421127 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.421131 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.421136 | controller |     - '        counter=0'
2026-03-14 09:44:27.421140 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.421144 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.421148 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.421153 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.421157 | controller |     - '            fi'
2026-03-14 09:44:27.421161 | controller |     - '        done'
2026-03-14 09:44:27.421165 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.421171 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.421175 | controller |     - '          exit 1'
2026-03-14 09:44:27.421180 | controller |     - '        else'
2026-03-14 09:44:27.421184 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.421188 | controller |     - '        fi'
2026-03-14 09:44:27.421192 | controller |     - '        ]'
2026-03-14 09:44:27.421197 | controller |     - '    logger.go:42: 09:39:49 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.421201 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.421205 | controller |     - '    logger.go:42: 09:39:49 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.421209 | controller |     - '    logger.go:42: 09:39:52 | watcher/1-deploy-with-defaults | + ''['' 0 == 1 '']'''
2026-03-14 09:44:27.421213 | controller |     - '    logger.go:42: 09:39:53 | watcher/1-deploy-with-defaults | running command:
2026-03-14 09:44:27.421218 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.421222 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.421226 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.421230 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:44:27.421235 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.421239 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.421243 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.421247 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.421251 | controller |       ]'
2026-03-14 09:44:27.421256 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.421260 | controller |       -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:44:27.421264 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.421268 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')"
2026-03-14 09:44:27.421273 | controller |       == 2 ]'
2026-03-14 09:44:27.421277 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.421281 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')"
2026-03-14 09:44:27.421286 | controller |       -ge 1 ]'
2026-03-14 09:44:27.421307 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.421315 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.421319 | controller |       == "" ]; then'
2026-03-14 09:44:27.421323 | controller |     - '            exit 0'
2026-03-14 09:44:27.421328 | controller |     - '        fi'
2026-03-14 09:44:27.421332 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.421336 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.421340 | controller |     - '        counter=0'
2026-03-14 09:44:27.421354 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.421359 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.421363 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.421367 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.421372 | controller |     - '            fi'
2026-03-14 09:44:27.421376 | controller |     - '        done'
2026-03-14 09:44:27.421380 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.421384 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.421389 | controller |     - '          exit 1'
2026-03-14 09:44:27.421393 | controller |     - '        else'
2026-03-14 09:44:27.421399 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.421403 | controller |     - '        fi'
2026-03-14 09:44:27.421407 | controller |     - '        ]'
2026-03-14 09:44:27.421411 | controller |     - '    logger.go:42: 09:39:53 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.421416 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.421420 | controller |     - '    logger.go:42: 09:39:53 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.421424 | controller |     - '    logger.go:42: 09:39:55 | watcher/1-deploy-with-defaults | + ''['' 0 == 1 '']'''
2026-03-14 09:44:27.421428 | controller |     - '    logger.go:42: 09:39:56 | watcher/1-deploy-with-defaults | running command:
2026-03-14 09:44:27.421433 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.421437 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.421441 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.421445 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:44:27.421449 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.421454 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.421458 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.421462 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.421466 | controller |       ]'
2026-03-14 09:44:27.421470 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.421474 | controller |       -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:44:27.421479 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.421483 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')"
2026-03-14 09:44:27.421487 | controller |       == 2 ]'
2026-03-14 09:44:27.421491 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.421495 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')"
2026-03-14 09:44:27.421500 | controller |       -ge 1 ]'
2026-03-14 09:44:27.421504 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.421508 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.421512 | controller |       == "" ]; then'
2026-03-14 09:44:27.421517 | controller |     - '            exit 0'
2026-03-14 09:44:27.421521 | controller |     - '        fi'
2026-03-14 09:44:27.421525 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.421529 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.421533 | controller |     - '        counter=0'
2026-03-14 09:44:27.421537 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.421542 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.421546 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.421550 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.421554 | controller |     - '            fi'
2026-03-14 09:44:27.421559 | controller |     - '        done'
2026-03-14 09:44:27.421563 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.421567 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.421571 | controller |     - '          exit 1'
2026-03-14 09:44:27.421575 | controller |     - '        else'
2026-03-14 09:44:27.421580 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.421584 | controller |     - '        fi'
2026-03-14 09:44:27.421588 | controller |     - '        ]'
2026-03-14 09:44:27.421592 | controller |     - '    logger.go:42: 09:39:56 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.421598 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.421603 | controller |     - '    logger.go:42: 09:39:56 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.421607 | controller |     - '    logger.go:42: 09:39:58 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.421613 | controller |     - '    logger.go:42: 09:39:58 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}'''
2026-03-14 09:44:27.421617 | controller |     - '    logger.go:42: 09:39:58 | watcher/1-deploy-with-defaults | ++ grep watcher'
2026-03-14 09:44:27.421621 | controller |     - '    logger.go:42: 09:39:58 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.421626 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:44:27.421630 | controller |     - '    logger.go:42: 09:40:00 | watcher/1-deploy-with-defaults | + SERVICEID=55dad13ac5da4c7fbfa109c6b4aa619f'
2026-03-14 09:44:27.421666 | controller |     - '    logger.go:42: 09:40:00 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.421676 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.421681 | controller |     - '    logger.go:42: 09:40:00 | watcher/1-deploy-with-defaults | + ''['' 55dad13ac5da4c7fbfa109c6b4aa619f
2026-03-14 09:44:27.421685 | controller |       == 55dad13ac5da4c7fbfa109c6b4aa619f '']'''
2026-03-14 09:44:27.421690 | controller |     - '    logger.go:42: 09:40:00 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.421694 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.421698 | controller |     - '    logger.go:42: 09:40:00 | watcher/1-deploy-with-defaults | + ''['' -n '''' '']'''
2026-03-14 09:44:27.421703 | controller |     - '    logger.go:42: 09:40:01 | watcher/1-deploy-with-defaults | running command:
2026-03-14 09:44:27.421707 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.421711 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.421715 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.421720 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:44:27.421734 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.421740 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.421746 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.421752 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.421759 | controller |       ]'
2026-03-14 09:44:27.421766 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.421780 | controller |       -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:44:27.421786 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.421790 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')"
2026-03-14 09:44:27.421795 | controller |       == 2 ]'
2026-03-14 09:44:27.421799 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.421803 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')"
2026-03-14 09:44:27.421811 | controller |       -ge 1 ]'
2026-03-14 09:44:27.421817 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.421822 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.421827 | controller |       == "" ]; then'
2026-03-14 09:44:27.421833 | controller |     - '            exit 0'
2026-03-14 09:44:27.421874 | controller |     - '        fi'
2026-03-14 09:44:27.421882 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.421891 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.421895 | controller |     - '        counter=0'
2026-03-14 09:44:27.421899 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.421903 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.421908 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.421912 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.421920 | controller |     - '            fi'
2026-03-14 09:44:27.421935 | controller |     - '        done'
2026-03-14 09:44:27.421940 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.421945 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.421953 | controller |     - '          exit 1'
2026-03-14 09:44:27.421967 | controller |     - '        else'
2026-03-14 09:44:27.421971 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.421975 | controller |     - '        fi'
2026-03-14 09:44:27.421980 | controller |     - '        ]'
2026-03-14 09:44:27.421984 | controller |     - '    logger.go:42: 09:40:01 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.421989 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.421995 | controller |     - '    logger.go:42: 09:40:01 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.422010 | controller |     - '    logger.go:42: 09:40:03 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.422014 | controller |     - '    logger.go:42: 09:40:03 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}'''
2026-03-14 09:44:27.422018 | controller |     - '    logger.go:42: 09:40:03 | watcher/1-deploy-with-defaults | ++ grep watcher'
2026-03-14 09:44:27.422025 | controller |     - '    logger.go:42: 09:40:03 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.422038 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:44:27.422042 | controller |     - '    logger.go:42: 09:40:05 | watcher/1-deploy-with-defaults | + SERVICEID=55dad13ac5da4c7fbfa109c6b4aa619f'
2026-03-14 09:44:27.422046 | controller |     - '    logger.go:42: 09:40:05 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.422053 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.422067 | controller |     - '    logger.go:42: 09:40:05 | watcher/1-deploy-with-defaults | + ''['' 55dad13ac5da4c7fbfa109c6b4aa619f
2026-03-14 09:44:27.422071 | controller |       == 55dad13ac5da4c7fbfa109c6b4aa619f '']'''
2026-03-14 09:44:27.422075 | controller |     - '    logger.go:42: 09:40:05 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.422083 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.422104 | controller |     - '    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | + ''['' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.422112 | controller |       '']'''
2026-03-14 09:44:27.422120 | controller |     - '    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:44:27.422126 | controller |     - '    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.422131 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}'''
2026-03-14 09:44:27.422144 | controller |     - '    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1'
2026-03-14 09:44:27.422159 | controller |     - '    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.422165 | controller |     - '    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:44:27.422235 | controller |     - '    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.422249 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:44:27.422259 | controller |     - '    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | ++ grep -c ''cafile
2026-03-14 09:44:27.422265 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'''
2026-03-14 09:44:27.422271 | controller |     - '    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | + ''['' 2 == 2 '']'''
2026-03-14 09:44:27.422276 | controller |     - '    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | ++ grep -c ''region_name
2026-03-14 09:44:27.422282 | controller |       = regionOne'''
2026-03-14 09:44:27.422288 | controller |     - '    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:44:27.422312 | controller |     - '    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.422318 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:44:27.422326 | controller |     - '    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | + ''['' 9 -ge 1 '']'''
2026-03-14 09:44:27.422366 | controller |     - '    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | ++ oc get pods -n
2026-03-14 09:44:27.422381 | controller |       openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:44:27.422386 | controller |     - '    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | + ''['' '''' == ''''
2026-03-14 09:44:27.422390 | controller |       '']'''
2026-03-14 09:44:27.422395 | controller |     - '    logger.go:42: 09:40:06 | watcher/1-deploy-with-defaults | + exit 0'
2026-03-14 09:44:27.422404 | controller |     - '    logger.go:42: 09:40:07 | watcher/1-deploy-with-defaults | running command:
2026-03-14 09:44:27.422412 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.422416 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.422421 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.422425 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:44:27.422429 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.422433 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.422438 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.422447 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.422461 | controller |       ]'
2026-03-14 09:44:27.422466 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.422471 | controller |       -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:44:27.422477 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.422491 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')"
2026-03-14 09:44:27.422496 | controller |       == 2 ]'
2026-03-14 09:44:27.422500 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.422504 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')"
2026-03-14 09:44:27.422511 | controller |       -ge 1 ]'
2026-03-14 09:44:27.422525 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.422529 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.422533 | controller |       == "" ]; then'
2026-03-14 09:44:27.422538 | controller |     - '            exit 0'
2026-03-14 09:44:27.422542 | controller |     - '        fi'
2026-03-14 09:44:27.422547 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.422551 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.422555 | controller |     - '        counter=0'
2026-03-14 09:44:27.422560 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.422566 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.422570 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.422575 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.422579 | controller |     - '            fi'
2026-03-14 09:44:27.422583 | controller |     - '        done'
2026-03-14 09:44:27.422587 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.422592 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.422596 | controller |     - '          exit 1'
2026-03-14 09:44:27.422608 | controller |     - '        else'
2026-03-14 09:44:27.422612 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.422617 | controller |     - '        fi'
2026-03-14 09:44:27.422621 | controller |     - '        ]'
2026-03-14 09:44:27.422625 | controller |     - '    logger.go:42: 09:40:07 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.422631 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.422636 | controller |     - '    logger.go:42: 09:40:07 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.422640 | controller |     - '    logger.go:42: 09:40:09 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.422644 | controller |     - '    logger.go:42: 09:40:09 | watcher/1-deploy-with-defaults | ++ grep watcher'
2026-03-14 09:44:27.422649 | controller |     - '    logger.go:42: 09:40:09 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.422653 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:44:27.422657 | controller |     - '    logger.go:42: 09:40:09 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}'''
2026-03-14 09:44:27.422662 | controller |     - '    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | + SERVICEID=55dad13ac5da4c7fbfa109c6b4aa619f'
2026-03-14 09:44:27.422666 | controller |     - '    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.422670 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.422674 | controller |     - '    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | + ''['' 55dad13ac5da4c7fbfa109c6b4aa619f
2026-03-14 09:44:27.422679 | controller |       == 55dad13ac5da4c7fbfa109c6b4aa619f '']'''
2026-03-14 09:44:27.422683 | controller |     - '    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.422687 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.422692 | controller |     - '    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | + ''['' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.422696 | controller |       '']'''
2026-03-14 09:44:27.422700 | controller |     - '    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:44:27.422705 | controller |     - '    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.422709 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}'''
2026-03-14 09:44:27.422713 | controller |     - '    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1'
2026-03-14 09:44:27.422717 | controller |     - '    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.422722 | controller |     - '    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | ++ grep -c ''cafile
2026-03-14 09:44:27.422726 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'''
2026-03-14 09:44:27.422730 | controller |     - '    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:44:27.422735 | controller |     - '    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.422739 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:44:27.422743 | controller |     - '    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | + ''['' 2 == 2 '']'''
2026-03-14 09:44:27.422750 | controller |     - '    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:44:27.422754 | controller |     - '    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.422759 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:44:27.422763 | controller |     - '    logger.go:42: 09:40:11 | watcher/1-deploy-with-defaults | ++ grep -c ''region_name
2026-03-14 09:44:27.422767 | controller |       = regionOne'''
2026-03-14 09:44:27.422771 | controller |     - '    logger.go:42: 09:40:12 | watcher/1-deploy-with-defaults | + ''['' 9 -ge 1 '']'''
2026-03-14 09:44:27.422776 | controller |     - '    logger.go:42: 09:40:12 | watcher/1-deploy-with-defaults | ++ oc get pods -n
2026-03-14 09:44:27.422780 | controller |       openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:44:27.422784 | controller |     - '    logger.go:42: 09:40:12 | watcher/1-deploy-with-defaults | + ''['' '''' == ''''
2026-03-14 09:44:27.422789 | controller |       '']'''
2026-03-14 09:44:27.422793 | controller |     - '    logger.go:42: 09:40:12 | watcher/1-deploy-with-defaults | + exit 0'
2026-03-14 09:44:27.422797 | controller |     - '    logger.go:42: 09:40:13 | watcher/1-deploy-with-defaults | running command:
2026-03-14 09:44:27.422801 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.422806 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.422810 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.422814 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:44:27.422819 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.422823 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.422827 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.422839 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.422844 | controller |       ]'
2026-03-14 09:44:27.422848 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.422853 | controller |       -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:44:27.422857 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.422861 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')"
2026-03-14 09:44:27.422866 | controller |       == 2 ]'
2026-03-14 09:44:27.422870 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.422874 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')"
2026-03-14 09:44:27.422879 | controller |       -ge 1 ]'
2026-03-14 09:44:27.422883 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.422887 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.422892 | controller |       == "" ]; then'
2026-03-14 09:44:27.422896 | controller |     - '            exit 0'
2026-03-14 09:44:27.422900 | controller |     - '        fi'
2026-03-14 09:44:27.422905 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.422909 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.422913 | controller |     - '        counter=0'
2026-03-14 09:44:27.422918 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.422922 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.422928 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.422934 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.422940 | controller |     - '            fi'
2026-03-14 09:44:27.422951 | controller |     - '        done'
2026-03-14 09:44:27.422958 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.422964 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.422969 | controller |     - '          exit 1'
2026-03-14 09:44:27.422975 | controller |     - '        else'
2026-03-14 09:44:27.422981 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.422987 | controller |     - '        fi'
2026-03-14 09:44:27.422993 | controller |     - '        ]'
2026-03-14 09:44:27.422999 | controller |     - '    logger.go:42: 09:40:13 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.423038 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.423044 | controller |     - '    logger.go:42: 09:40:13 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.423049 | controller |     - '    logger.go:42: 09:40:15 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.423053 | controller |     - '    logger.go:42: 09:40:15 | watcher/1-deploy-with-defaults | ++ grep watcher'
2026-03-14 09:44:27.423057 | controller |     - '    logger.go:42: 09:40:15 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.423061 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:44:27.423066 | controller |     - '    logger.go:42: 09:40:15 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}'''
2026-03-14 09:44:27.423070 | controller |     - '    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | + SERVICEID=55dad13ac5da4c7fbfa109c6b4aa619f'
2026-03-14 09:44:27.423074 | controller |     - '    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.423079 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.423083 | controller |     - '    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | + ''['' 55dad13ac5da4c7fbfa109c6b4aa619f
2026-03-14 09:44:27.423087 | controller |       == 55dad13ac5da4c7fbfa109c6b4aa619f '']'''
2026-03-14 09:44:27.423091 | controller |     - '    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.423096 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.423100 | controller |     - '    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | + ''['' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.423104 | controller |       '']'''
2026-03-14 09:44:27.423109 | controller |     - '    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:44:27.423113 | controller |     - '    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.423117 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}'''
2026-03-14 09:44:27.423121 | controller |     - '    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1'
2026-03-14 09:44:27.423125 | controller |     - '    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.423130 | controller |     - '    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:44:27.423134 | controller |     - '    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.423138 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:44:27.423142 | controller |     - '    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | ++ grep -c ''cafile
2026-03-14 09:44:27.423147 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'''
2026-03-14 09:44:27.423159 | controller |     - '    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | + ''['' 2 == 2 '']'''
2026-03-14 09:44:27.423163 | controller |     - '    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:44:27.423168 | controller |     - '    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.423172 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:44:27.423179 | controller |     - '    logger.go:42: 09:40:17 | watcher/1-deploy-with-defaults | ++ grep -c ''region_name
2026-03-14 09:44:27.423183 | controller |       = regionOne'''
2026-03-14 09:44:27.423187 | controller |     - '    logger.go:42: 09:40:18 | watcher/1-deploy-with-defaults | + ''['' 9 -ge 1 '']'''
2026-03-14 09:44:27.423192 | controller |     - '    logger.go:42: 09:40:18 | watcher/1-deploy-with-defaults | ++ oc get pods -n
2026-03-14 09:44:27.423196 | controller |       openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:44:27.423200 | controller |     - '    logger.go:42: 09:40:18 | watcher/1-deploy-with-defaults | + ''['' '''' == ''''
2026-03-14 09:44:27.423204 | controller |       '']'''
2026-03-14 09:44:27.423209 | controller |     - '    logger.go:42: 09:40:18 | watcher/1-deploy-with-defaults | + exit 0'
2026-03-14 09:44:27.423213 | controller |     - '    logger.go:42: 09:40:19 | watcher/1-deploy-with-defaults | running command:
2026-03-14 09:44:27.423217 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.423222 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.423226 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.423230 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:44:27.423234 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.423239 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.423243 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.423247 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.423251 | controller |       ]'
2026-03-14 09:44:27.423256 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.423260 | controller |       -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]'
2026-03-14 09:44:27.423264 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.423269 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')"
2026-03-14 09:44:27.423273 | controller |       == 2 ]'
2026-03-14 09:44:27.423277 | controller |     - '        [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data
2026-03-14 09:44:27.423281 | controller |       -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')"
2026-03-14 09:44:27.423286 | controller |       -ge 1 ]'
2026-03-14 09:44:27.423309 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.423317 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.423321 | controller |       == "" ]; then'
2026-03-14 09:44:27.423326 | controller |     - '            exit 0'
2026-03-14 09:44:27.423330 | controller |     - '        fi'
2026-03-14 09:44:27.423337 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.423341 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.423345 | controller |     - '        counter=0'
2026-03-14 09:44:27.423349 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.423353 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.423358 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.423362 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.423366 | controller |     - '            fi'
2026-03-14 09:44:27.423370 | controller |     - '        done'
2026-03-14 09:44:27.423375 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.423379 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.423383 | controller |     - '          exit 1'
2026-03-14 09:44:27.423388 | controller |     - '        else'
2026-03-14 09:44:27.423395 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.423399 | controller |     - '        fi'
2026-03-14 09:44:27.423405 | controller |     - '        ]'
2026-03-14 09:44:27.423409 | controller |     - '    logger.go:42: 09:40:19 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.423414 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.423418 | controller |     - '    logger.go:42: 09:40:19 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.423422 | controller |     - '    logger.go:42: 09:40:21 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.423426 | controller |     - '    logger.go:42: 09:40:21 | watcher/1-deploy-with-defaults | ++ grep watcher'
2026-03-14 09:44:27.423431 | controller |     - '    logger.go:42: 09:40:21 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.423435 | controller |       openstackclient -- openstack service list -f value -c Name -c Type -c ID'
2026-03-14 09:44:27.423439 | controller |     - '    logger.go:42: 09:40:21 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}'''
2026-03-14 09:44:27.423443 | controller |     - '    logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults | + SERVICEID=55dad13ac5da4c7fbfa109c6b4aa619f'
2026-03-14 09:44:27.423448 | controller |     - '    logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.423459 | controller |       keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.423464 | controller |     - '    logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults | + ''['' 55dad13ac5da4c7fbfa109c6b4aa619f
2026-03-14 09:44:27.423468 | controller |       == 55dad13ac5da4c7fbfa109c6b4aa619f '']'''
2026-03-14 09:44:27.423474 | controller |     - '    logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.423479 | controller |       watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.423483 | controller |     - '    logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults | + ''['' -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.423487 | controller |       '']'''
2026-03-14 09:44:27.423492 | controller |     - '    logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:44:27.423496 | controller |     - '    logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.423500 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}'''
2026-03-14 09:44:27.423505 | controller |     - '    logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1'
2026-03-14 09:44:27.423509 | controller |     - '    logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']'''
2026-03-14 09:44:27.423513 | controller |     - '    logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:44:27.423518 | controller |     - '    logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.423522 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:44:27.423526 | controller |     - '    logger.go:42: 09:40:23 | watcher/1-deploy-with-defaults | ++ grep -c ''cafile
2026-03-14 09:44:27.423530 | controller |       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'''
2026-03-14 09:44:27.423548 | controller |     - '    logger.go:42: 09:40:24 | watcher/1-deploy-with-defaults | + ''['' 2 == 2 '']'''
2026-03-14 09:44:27.423553 | controller |     - '    logger.go:42: 09:40:24 | watcher/1-deploy-with-defaults | ++ grep -c ''region_name
2026-03-14 09:44:27.423558 | controller |       = regionOne'''
2026-03-14 09:44:27.423562 | controller |     - '    logger.go:42: 09:40:24 | watcher/1-deploy-with-defaults | ++ base64 -d'
2026-03-14 09:44:27.423566 | controller |     - '    logger.go:42: 09:40:24 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default
2026-03-14 09:44:27.423571 | controller |       secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}'''
2026-03-14 09:44:27.423575 | controller |     - '    logger.go:42: 09:40:24 | watcher/1-deploy-with-defaults | + ''['' 9 -ge 1 '']'''
2026-03-14 09:44:27.423581 | controller |     - '    logger.go:42: 09:40:24 | watcher/1-deploy-with-defaults | ++ oc get pods -n
2026-03-14 09:44:27.423586 | controller |       openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:44:27.423590 | controller |     - '    logger.go:42: 09:40:24 | watcher/1-deploy-with-defaults | + ''['' '''' == ''''
2026-03-14 09:44:27.423594 | controller |       '']'''
2026-03-14 09:44:27.423599 | controller |     - '    logger.go:42: 09:40:24 | watcher/1-deploy-with-defaults | + exit 0'
2026-03-14 09:44:27.423603 | controller |     - '    logger.go:42: 09:40:24 | watcher/1-deploy-with-defaults | test step completed
2026-03-14 09:44:27.423607 | controller |       1-deploy-with-defaults'
2026-03-14 09:44:27.423612 | controller |     - '    logger.go:42: 09:40:24 | watcher/2-cleanup-watcher | starting test step 2-cleanup-watcher'
2026-03-14 09:44:27.423616 | controller |     - '    logger.go:42: 09:40:24 | watcher/2-cleanup-watcher | test step completed 2-cleanup-watcher'
2026-03-14 09:44:27.423620 | controller |     - '    logger.go:42: 09:40:24 | watcher/3-precreate-mariadbaccount | starting test
2026-03-14 09:44:27.423624 | controller |       step 3-precreate-mariadbaccount'
2026-03-14 09:44:27.423629 | controller |     - '    logger.go:42: 09:40:24 | watcher/3-precreate-mariadbaccount | MariaDBAccount:watcher-kuttl-default/watcher-precreated
2026-03-14 09:44:27.423633 | controller |       created'
2026-03-14 09:44:27.423637 | controller |     - '    logger.go:42: 09:40:24 | watcher/3-precreate-mariadbaccount | test step completed
2026-03-14 09:44:27.423642 | controller |       3-precreate-mariadbaccount'
2026-03-14 09:44:27.423646 | controller |     - '    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account | starting
2026-03-14 09:44:27.423650 | controller |       test step 4-deploy-with-precreated-account'
2026-03-14 09:44:27.423655 | controller |     - '    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account | Secret:watcher-kuttl-default/custom-prometheus-config
2026-03-14 09:44:27.423659 | controller |       created'
2026-03-14 09:44:27.423663 | controller |     - '    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:44:27.423667 | controller |       created'
2026-03-14 09:44:27.423672 | controller |     - '    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.423676 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.423680 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.423685 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.423689 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.423693 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.423698 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.423702 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.423715 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.423720 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.423726 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.423731 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.423735 | controller |       == 1 ]'
2026-03-14 09:44:27.423739 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.423744 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.423748 | controller |     - '        else'
2026-03-14 09:44:27.423753 | controller |     - '          exit 1'
2026-03-14 09:44:27.423757 | controller |     - '        fi'
2026-03-14 09:44:27.423761 | controller |     - '        ]'
2026-03-14 09:44:27.423766 | controller |     - '    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.423770 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.423776 | controller |     - '    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.423780 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.423784 | controller |     - '    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.423789 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.423793 | controller |     - '    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.423797 | controller |       -1'
2026-03-14 09:44:27.423804 | controller |     - '    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.423809 | controller |       -v ''^$'''
2026-03-14 09:44:27.423813 | controller |     - '    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.423817 | controller |     - '    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.423821 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.423826 | controller |     - '    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.423830 | controller |       -c ''^# Global config'''
2026-03-14 09:44:27.423834 | controller |     - '    logger.go:42: 09:40:24 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.423839 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:44:27.423843 | controller |     - '    logger.go:42: 09:40:25 | watcher/4-deploy-with-precreated-account | error:
2026-03-14 09:44:27.423847 | controller |       Internal error occurred: error executing command in container: container is not
2026-03-14 09:44:27.423851 | controller |       created or running'
2026-03-14 09:44:27.423856 | controller |     - '    logger.go:42: 09:40:25 | watcher/4-deploy-with-precreated-account | ++ echo'
2026-03-14 09:44:27.423860 | controller |     - '    logger.go:42: 09:40:25 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.423864 | controller |       0 == 1 '']'''
2026-03-14 09:44:27.423868 | controller |     - '    logger.go:42: 09:40:29 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.423872 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.423877 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.423881 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.423885 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.423889 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.423894 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.423898 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.423902 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.423906 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.423911 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.423915 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.423920 | controller |       == 1 ]'
2026-03-14 09:44:27.423925 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.423930 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.423934 | controller |     - '        else'
2026-03-14 09:44:27.423939 | controller |     - '          exit 1'
2026-03-14 09:44:27.423944 | controller |     - '        fi'
2026-03-14 09:44:27.423949 | controller |     - '        ]'
2026-03-14 09:44:27.423954 | controller |     - '    logger.go:42: 09:40:29 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.423958 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.423964 | controller |     - '    logger.go:42: 09:40:29 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.423968 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.423972 | controller |     - '    logger.go:42: 09:40:29 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.423976 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.423981 | controller |     - '    logger.go:42: 09:40:29 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.423985 | controller |       -1'
2026-03-14 09:44:27.423995 | controller |     - '    logger.go:42: 09:40:29 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.424001 | controller |       -v ''^$'''
2026-03-14 09:44:27.424005 | controller |     - '    logger.go:42: 09:40:29 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:44:27.424009 | controller |     - '    logger.go:42: 09:40:30 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.424013 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.424018 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.424022 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.424026 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.424030 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.424035 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.424039 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.424043 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.424047 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.424052 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.424056 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.424060 | controller |       == 1 ]'
2026-03-14 09:44:27.424064 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.424069 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.424073 | controller |     - '        else'
2026-03-14 09:44:27.424077 | controller |     - '          exit 1'
2026-03-14 09:44:27.424081 | controller |     - '        fi'
2026-03-14 09:44:27.424086 | controller |     - '        ]'
2026-03-14 09:44:27.424090 | controller |     - '    logger.go:42: 09:40:30 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.424094 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.424098 | controller |     - '    logger.go:42: 09:40:30 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.424103 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.424107 | controller |     - '    logger.go:42: 09:40:30 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.424111 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.424115 | controller |     - '    logger.go:42: 09:40:30 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.424119 | controller |       -v ''^$'''
2026-03-14 09:44:27.424124 | controller |     - '    logger.go:42: 09:40:30 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.424128 | controller |       -1'
2026-03-14 09:44:27.424132 | controller |     - '    logger.go:42: 09:40:30 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:44:27.424137 | controller |     - '    logger.go:42: 09:40:31 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.424141 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.424145 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.424149 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.424155 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.424159 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.424163 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.424168 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.424172 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.424176 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.424180 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.424185 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.424189 | controller |       == 1 ]'
2026-03-14 09:44:27.424193 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.424197 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.424201 | controller |     - '        else'
2026-03-14 09:44:27.424206 | controller |     - '          exit 1'
2026-03-14 09:44:27.424210 | controller |     - '        fi'
2026-03-14 09:44:27.424214 | controller |     - '        ]'
2026-03-14 09:44:27.424218 | controller |     - '    logger.go:42: 09:40:31 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.424222 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.424228 | controller |     - '    logger.go:42: 09:40:31 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.424233 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.424237 | controller |     - '    logger.go:42: 09:40:31 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.424241 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.424245 | controller |     - '    logger.go:42: 09:40:31 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.424250 | controller |       -1'
2026-03-14 09:44:27.424260 | controller |     - '    logger.go:42: 09:40:31 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.424264 | controller |       -v ''^$'''
2026-03-14 09:44:27.424269 | controller |     - '    logger.go:42: 09:40:32 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:44:27.424273 | controller |     - '    logger.go:42: 09:40:33 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.424277 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.424281 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.424286 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.424303 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.424310 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.424314 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.424319 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.424323 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.424327 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.424331 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.424336 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.424340 | controller |       == 1 ]'
2026-03-14 09:44:27.424344 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.424350 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.424355 | controller |     - '        else'
2026-03-14 09:44:27.424359 | controller |     - '          exit 1'
2026-03-14 09:44:27.424363 | controller |     - '        fi'
2026-03-14 09:44:27.424367 | controller |     - '        ]'
2026-03-14 09:44:27.424373 | controller |     - '    logger.go:42: 09:40:33 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.424379 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.424385 | controller |     - '    logger.go:42: 09:40:33 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.424391 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.424397 | controller |     - '    logger.go:42: 09:40:33 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.424403 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.424408 | controller |     - '    logger.go:42: 09:40:33 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.424414 | controller |       -1'
2026-03-14 09:44:27.424420 | controller |     - '    logger.go:42: 09:40:33 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.424426 | controller |       -v ''^$'''
2026-03-14 09:44:27.424432 | controller |     - '    logger.go:42: 09:40:33 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:44:27.424437 | controller |     - '    logger.go:42: 09:40:34 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.424441 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.424445 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.424449 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.424454 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.424458 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.424462 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.424466 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.424471 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.424475 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.424479 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.424484 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.424490 | controller |       == 1 ]'
2026-03-14 09:44:27.424496 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.424502 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.424508 | controller |     - '        else'
2026-03-14 09:44:27.424513 | controller |     - '          exit 1'
2026-03-14 09:44:27.424519 | controller |     - '        fi'
2026-03-14 09:44:27.424525 | controller |     - '        ]'
2026-03-14 09:44:27.424531 | controller |     - '    logger.go:42: 09:40:34 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.424537 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.424541 | controller |     - '    logger.go:42: 09:40:34 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.424546 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.424550 | controller |     - '    logger.go:42: 09:40:34 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.424554 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.424558 | controller |     - '    logger.go:42: 09:40:34 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.424563 | controller |       -1'
2026-03-14 09:44:27.424567 | controller |     - '    logger.go:42: 09:40:34 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.424575 | controller |       -v ''^$'''
2026-03-14 09:44:27.424588 | controller |     - '    logger.go:42: 09:40:34 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:44:27.424592 | controller |     - '    logger.go:42: 09:40:35 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.424597 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.424601 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.424605 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.424610 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.424614 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.424618 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.424623 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.424627 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.424631 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.424635 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.424639 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.424644 | controller |       == 1 ]'
2026-03-14 09:44:27.424648 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.424652 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.424656 | controller |     - '        else'
2026-03-14 09:44:27.424661 | controller |     - '          exit 1'
2026-03-14 09:44:27.424665 | controller |     - '        fi'
2026-03-14 09:44:27.424669 | controller |     - '        ]'
2026-03-14 09:44:27.424673 | controller |     - '    logger.go:42: 09:40:35 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.424678 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.424682 | controller |     - '    logger.go:42: 09:40:36 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.424689 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.424694 | controller |     - '    logger.go:42: 09:40:36 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.424698 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.424702 | controller |     - '    logger.go:42: 09:40:36 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.424706 | controller |       -1'
2026-03-14 09:44:27.424710 | controller |     - '    logger.go:42: 09:40:36 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.424715 | controller |       -v ''^$'''
2026-03-14 09:44:27.424719 | controller |     - '    logger.go:42: 09:40:36 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:44:27.424723 | controller |     - '    logger.go:42: 09:40:37 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.424727 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.424731 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.424736 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.424740 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.424744 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.424748 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.424753 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.424757 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.424761 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.424765 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.424771 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.424775 | controller |       == 1 ]'
2026-03-14 09:44:27.424780 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.424784 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.424788 | controller |     - '        else'
2026-03-14 09:44:27.424792 | controller |     - '          exit 1'
2026-03-14 09:44:27.424797 | controller |     - '        fi'
2026-03-14 09:44:27.424801 | controller |     - '        ]'
2026-03-14 09:44:27.424805 | controller |     - '    logger.go:42: 09:40:37 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.424809 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.424813 | controller |     - '    logger.go:42: 09:40:37 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.424818 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.424822 | controller |     - '    logger.go:42: 09:40:37 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.424826 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.424830 | controller |     - '    logger.go:42: 09:40:37 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.424835 | controller |       -1'
2026-03-14 09:44:27.424839 | controller |     - '    logger.go:42: 09:40:37 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.424843 | controller |       -v ''^$'''
2026-03-14 09:44:27.424847 | controller |     - '    logger.go:42: 09:40:37 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:44:27.424857 | controller |     - '    logger.go:42: 09:40:38 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.424863 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.424868 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.424874 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.424880 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.424885 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.424890 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.424896 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.424916 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.424923 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.424928 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.424932 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.424936 | controller |       == 1 ]'
2026-03-14 09:44:27.424941 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.424945 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.424949 | controller |     - '        else'
2026-03-14 09:44:27.424954 | controller |     - '          exit 1'
2026-03-14 09:44:27.424960 | controller |     - '        fi'
2026-03-14 09:44:27.424966 | controller |     - '        ]'
2026-03-14 09:44:27.424972 | controller |     - '    logger.go:42: 09:40:38 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.424978 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.424985 | controller |     - '    logger.go:42: 09:40:38 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.424991 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.424997 | controller |     - '    logger.go:42: 09:40:38 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.425006 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.425012 | controller |     - '    logger.go:42: 09:40:38 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.425018 | controller |       -1'
2026-03-14 09:44:27.425024 | controller |     - '    logger.go:42: 09:40:38 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.425030 | controller |       -v ''^$'''
2026-03-14 09:44:27.425036 | controller |     - '    logger.go:42: 09:40:38 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:44:27.425041 | controller |     - '    logger.go:42: 09:40:39 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.425047 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.425053 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.425058 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.425064 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.425069 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.425075 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.425080 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.425086 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.425091 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.425095 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.425099 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.425103 | controller |       == 1 ]'
2026-03-14 09:44:27.425107 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.425112 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.425116 | controller |     - '        else'
2026-03-14 09:44:27.425120 | controller |     - '          exit 1'
2026-03-14 09:44:27.425124 | controller |     - '        fi'
2026-03-14 09:44:27.425131 | controller |     - '        ]'
2026-03-14 09:44:27.425136 | controller |     - '    logger.go:42: 09:40:39 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.425141 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.425146 | controller |     - '    logger.go:42: 09:40:40 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.425152 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.425159 | controller |     - '    logger.go:42: 09:40:40 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.425165 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.425171 | controller |     - '    logger.go:42: 09:40:40 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.425176 | controller |       -1'
2026-03-14 09:44:27.425182 | controller |     - '    logger.go:42: 09:40:40 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.425187 | controller |       -v ''^$'''
2026-03-14 09:44:27.425193 | controller |     - '    logger.go:42: 09:40:40 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:44:27.425209 | controller |     - '    logger.go:42: 09:40:41 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.425215 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.425221 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.425227 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.425232 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.425237 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.425242 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.425250 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.425258 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.425264 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.425271 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.425278 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.425284 | controller |       == 1 ]'
2026-03-14 09:44:27.425305 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.425315 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.425321 | controller |     - '        else'
2026-03-14 09:44:27.425327 | controller |     - '          exit 1'
2026-03-14 09:44:27.425332 | controller |     - '        fi'
2026-03-14 09:44:27.425338 | controller |     - '        ]'
2026-03-14 09:44:27.425344 | controller |     - '    logger.go:42: 09:40:41 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.425349 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.425355 | controller |     - '    logger.go:42: 09:40:41 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.425361 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.425366 | controller |     - '    logger.go:42: 09:40:41 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.425372 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.425378 | controller |     - '    logger.go:42: 09:40:41 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.425384 | controller |       -1'
2026-03-14 09:44:27.425389 | controller |     - '    logger.go:42: 09:40:41 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.425395 | controller |       -v ''^$'''
2026-03-14 09:44:27.425400 | controller |     - '    logger.go:42: 09:40:41 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:44:27.425406 | controller |     - '    logger.go:42: 09:40:42 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.425412 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.425417 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.425423 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.425428 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.425434 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.425440 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.425446 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.425451 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.425456 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.425462 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.425467 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.425473 | controller |       == 1 ]'
2026-03-14 09:44:27.425479 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.425484 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.425490 | controller |     - '        else'
2026-03-14 09:44:27.425496 | controller |     - '          exit 1'
2026-03-14 09:44:27.425501 | controller |     - '        fi'
2026-03-14 09:44:27.425506 | controller |     - '        ]'
2026-03-14 09:44:27.425514 | controller |     - '    logger.go:42: 09:40:42 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.425520 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.425525 | controller |     - '    logger.go:42: 09:40:42 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.425530 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.425536 | controller |     - '    logger.go:42: 09:40:42 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.425542 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.425547 | controller |     - '    logger.go:42: 09:40:42 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.425552 | controller |       -1'
2026-03-14 09:44:27.425558 | controller |     - '    logger.go:42: 09:40:42 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.425564 | controller |       -v ''^$'''
2026-03-14 09:44:27.425570 | controller |     - '    logger.go:42: 09:40:42 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:44:27.425575 | controller |     - '    logger.go:42: 09:40:44 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.425581 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.425596 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.425602 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.425608 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.425613 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.425619 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.425625 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.425631 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.425636 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.425641 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.425647 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.425652 | controller |       == 1 ]'
2026-03-14 09:44:27.425657 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.425662 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.425668 | controller |     - '        else'
2026-03-14 09:44:27.425674 | controller |     - '          exit 1'
2026-03-14 09:44:27.425679 | controller |     - '        fi'
2026-03-14 09:44:27.425684 | controller |     - '        ]'
2026-03-14 09:44:27.425689 | controller |     - '    logger.go:42: 09:40:44 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.425695 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.425700 | controller |     - '    logger.go:42: 09:40:44 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.425706 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.425711 | controller |     - '    logger.go:42: 09:40:44 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.425721 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.425726 | controller |     - '    logger.go:42: 09:40:44 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.425732 | controller |       -1'
2026-03-14 09:44:27.425738 | controller |     - '    logger.go:42: 09:40:44 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.425744 | controller |       -v ''^$'''
2026-03-14 09:44:27.425749 | controller |     - '    logger.go:42: 09:40:44 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:44:27.425755 | controller |     - '    logger.go:42: 09:40:45 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.425763 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.425769 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.425774 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.425779 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.425784 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.425789 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.425795 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.425800 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.425806 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.425811 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.425819 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.425824 | controller |       == 1 ]'
2026-03-14 09:44:27.425829 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.425834 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.425840 | controller |     - '        else'
2026-03-14 09:44:27.425845 | controller |     - '          exit 1'
2026-03-14 09:44:27.425850 | controller |     - '        fi'
2026-03-14 09:44:27.425855 | controller |     - '        ]'
2026-03-14 09:44:27.425860 | controller |     - '    logger.go:42: 09:40:45 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.425865 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.425870 | controller |     - '    logger.go:42: 09:40:45 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.425876 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.425881 | controller |     - '    logger.go:42: 09:40:45 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.425886 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.425891 | controller |     - '    logger.go:42: 09:40:45 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.425896 | controller |       -1'
2026-03-14 09:44:27.425902 | controller |     - '    logger.go:42: 09:40:45 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.425907 | controller |       -v ''^$'''
2026-03-14 09:44:27.425912 | controller |     - '    logger.go:42: 09:40:46 | watcher/4-deploy-with-precreated-account | + APIPOD='
2026-03-14 09:44:27.425917 | controller |     - '    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.425922 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.425928 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.425942 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.425949 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.425954 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.425959 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.425965 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.425970 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.425975 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.425980 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.425986 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.425995 | controller |       == 1 ]'
2026-03-14 09:44:27.426001 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.426006 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.426011 | controller |     - '        else'
2026-03-14 09:44:27.426019 | controller |     - '          exit 1'
2026-03-14 09:44:27.426025 | controller |     - '        fi'
2026-03-14 09:44:27.426030 | controller |     - '        ]'
2026-03-14 09:44:27.426035 | controller |     - '    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.426041 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.426046 | controller |     - '    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.426052 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.426057 | controller |     - '    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.426063 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.426069 | controller |     - '    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.426074 | controller |       -1'
2026-03-14 09:44:27.426080 | controller |     - '    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.426085 | controller |       -v ''^$'''
2026-03-14 09:44:27.426091 | controller |     - '    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.426096 | controller |     - '    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.426101 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.426106 | controller |     - '    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.426111 | controller |       -c ''^# Global config'''
2026-03-14 09:44:27.426117 | controller |     - '    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.426122 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:44:27.426128 | controller |     - '    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account | error:
2026-03-14 09:44:27.426133 | controller |       unable to upgrade connection: container not found ("watcher-api")'
2026-03-14 09:44:27.426139 | controller |     - '    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account | ++ echo'
2026-03-14 09:44:27.426145 | controller |     - '    logger.go:42: 09:40:47 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.426150 | controller |       0 == 1 '']'''
2026-03-14 09:44:27.426156 | controller |     - '    logger.go:42: 09:40:48 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.426162 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.426167 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.426173 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.426178 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.426210 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.426217 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.426223 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.426229 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.426235 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.426241 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.426247 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.426252 | controller |       == 1 ]'
2026-03-14 09:44:27.426258 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.426266 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.426272 | controller |     - '        else'
2026-03-14 09:44:27.426278 | controller |     - '          exit 1'
2026-03-14 09:44:27.426284 | controller |     - '        fi'
2026-03-14 09:44:27.426306 | controller |     - '        ]'
2026-03-14 09:44:27.426317 | controller |     - '    logger.go:42: 09:40:48 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.426323 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.426328 | controller |     - '    logger.go:42: 09:40:48 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.426344 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.426350 | controller |     - '    logger.go:42: 09:40:48 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.426356 | controller |       -v ''^$'''
2026-03-14 09:44:27.426362 | controller |     - '    logger.go:42: 09:40:48 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.426367 | controller |       -1'
2026-03-14 09:44:27.426373 | controller |     - '    logger.go:42: 09:40:48 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.426379 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.426384 | controller |     - '    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.426390 | controller |     - '    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.426396 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.426401 | controller |     - '    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.426407 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:44:27.426413 | controller |     - '    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.426418 | controller |       -c ''^# Global config'''
2026-03-14 09:44:27.426424 | controller |     - '    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.426430 | controller |       ''#'' Global config'
2026-03-14 09:44:27.426436 | controller |     - '    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.426441 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.426447 | controller |     - '    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.426453 | controller |       -c ''^# Service config'''
2026-03-14 09:44:27.426459 | controller |     - '    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.426465 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf'
2026-03-14 09:44:27.426471 | controller |     - '    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.426476 | controller |       ''#'' Service config'
2026-03-14 09:44:27.426482 | controller |     - '    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.426488 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.426494 | controller |     - '    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.426500 | controller |       -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'''
2026-03-14 09:44:27.426505 | controller |     - '    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.426511 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.426517 | controller |     - '    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.426522 | controller |       ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.426531 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.426537 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.426543 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_test:8df261d072f382ca7aeb59823b1f461f@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.426548 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.426554 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.426560 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.426565 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.426571 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.426577 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.426583 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.426588 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.426594 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.426599 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.426605 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.426611 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.426616 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.426633 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.426640 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.426647 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.426652 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.426657 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.426663 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.426668 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]''
2026-03-14 09:44:27.426677 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.426683 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.426688 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.426694 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.426700 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.426706 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.426712 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.426718 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.426724 | controller |     - '    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.426732 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.426738 | controller |     - '    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.426744 | controller |       -czPo ''TimeOut 80'''
2026-03-14 09:44:27.426749 | controller |     - '    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.426755 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf'
2026-03-14 09:44:27.426761 | controller |     - '    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.426769 | controller |       ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:44:27.426774 | controller |       ''*:9322>'' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:44:27.426780 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.426786 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.426792 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.426798 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:44:27.426804 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:44:27.426809 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:44:27.426816 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:44:27.426822 | controller |       internal display-name=internal group=watcher processes=2 threads=1 user=watcher
2026-03-14 09:44:27.426829 | controller |       WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>''
2026-03-14 09:44:27.426835 | controller |       ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:44:27.426841 | controller |       ''*:9322>'' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:44:27.426847 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.426853 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.426859 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.426865 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:44:27.426871 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:44:27.426877 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:44:27.426883 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:44:27.426889 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:44:27.426895 | controller |       public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>'''
2026-03-14 09:44:27.426901 | controller |     - '    logger.go:42: 09:40:49 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.426907 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.426923 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.426930 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.426936 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.426942 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.426948 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.426954 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.426960 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.426968 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.426974 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.426980 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.426986 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.426992 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.426998 | controller |       == 1 ]'
2026-03-14 09:44:27.427004 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.427009 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.427015 | controller |     - '        else'
2026-03-14 09:44:27.427021 | controller |     - '          exit 1'
2026-03-14 09:44:27.427027 | controller |     - '        fi'
2026-03-14 09:44:27.427033 | controller |     - '        ]'
2026-03-14 09:44:27.427039 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.427045 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.427050 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.427060 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.427066 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.427072 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.427078 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.427083 | controller |       -1'
2026-03-14 09:44:27.427089 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.427095 | controller |       -v ''^$'''
2026-03-14 09:44:27.427101 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.427107 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.427112 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.427118 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.427124 | controller |       -c ''^# Global config'''
2026-03-14 09:44:27.427130 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.427136 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:44:27.427141 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.427147 | controller |       ''#'' Global config'
2026-03-14 09:44:27.427153 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.427159 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.427165 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.427171 | controller |       -c ''^# Service config'''
2026-03-14 09:44:27.427177 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.427183 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf'
2026-03-14 09:44:27.427189 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.427195 | controller |       ''#'' Service config'
2026-03-14 09:44:27.427201 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.427207 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.427212 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.427218 | controller |       -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'''
2026-03-14 09:44:27.427227 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.427232 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.427238 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.427243 | controller |       ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.427248 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.427263 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.427269 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_test:8df261d072f382ca7aeb59823b1f461f@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.427274 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.427279 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.427285 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.427309 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.427320 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.427326 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.427332 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.427338 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.427344 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.427350 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.427356 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.427361 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.427367 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.427373 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.427378 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.427384 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.427390 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.427395 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.427401 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.427407 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]''
2026-03-14 09:44:27.427412 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.427418 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.427426 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.427432 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.427440 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.427446 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.427452 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.427457 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.427463 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.427469 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.427475 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.427481 | controller |       -czPo ''TimeOut 80'''
2026-03-14 09:44:27.427487 | controller |     - '    logger.go:42: 09:40:51 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.427493 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf'
2026-03-14 09:44:27.427499 | controller |     - '    logger.go:42: 09:40:52 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.427505 | controller |       ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:44:27.427514 | controller |       ''*:9322>'' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:44:27.427520 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.427525 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.427531 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.427537 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:44:27.427544 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:44:27.427560 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:44:27.427567 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:44:27.427574 | controller |       internal display-name=internal group=watcher processes=2 threads=1 user=watcher
2026-03-14 09:44:27.427580 | controller |       WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>''
2026-03-14 09:44:27.427585 | controller |       ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:44:27.427591 | controller |       ''*:9322>'' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:44:27.427597 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.427603 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.427609 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.427615 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:44:27.427621 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:44:27.427627 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:44:27.427632 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:44:27.427638 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:44:27.427644 | controller |       public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>'''
2026-03-14 09:44:27.427650 | controller |     - '    logger.go:42: 09:40:52 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.427656 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.427665 | controller |     - '    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.427672 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.427678 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.427684 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.427690 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.427696 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.427702 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.427709 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.427715 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.427721 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.427727 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.427740 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.427747 | controller |       == 1 ]'
2026-03-14 09:44:27.427754 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.427760 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.427766 | controller |     - '        else'
2026-03-14 09:44:27.427772 | controller |     - '          exit 1'
2026-03-14 09:44:27.427778 | controller |     - '        fi'
2026-03-14 09:44:27.427784 | controller |     - '        ]'
2026-03-14 09:44:27.427790 | controller |     - '    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.427797 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.427803 | controller |     - '    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.427809 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.427815 | controller |     - '    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.427821 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.427827 | controller |     - '    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.427834 | controller |       -1'
2026-03-14 09:44:27.427840 | controller |     - '    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.427846 | controller |       -v ''^$'''
2026-03-14 09:44:27.427853 | controller |     - '    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.427861 | controller |     - '    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.427868 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.427874 | controller |     - '    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.427879 | controller |       -c ''^# Global config'''
2026-03-14 09:44:27.427885 | controller |     - '    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.427891 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:44:27.427897 | controller |     - '    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.427903 | controller |       ''#'' Global config'
2026-03-14 09:44:27.427909 | controller |     - '    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.427926 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.427934 | controller |     - '    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.427940 | controller |       -c ''^# Service config'''
2026-03-14 09:44:27.427947 | controller |     - '    logger.go:42: 09:40:53 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.427956 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf'
2026-03-14 09:44:27.427962 | controller |     - '    logger.go:42: 09:40:54 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.427969 | controller |       ''#'' Service config'
2026-03-14 09:44:27.427975 | controller |     - '    logger.go:42: 09:40:54 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.427981 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.427987 | controller |     - '    logger.go:42: 09:40:54 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.427993 | controller |       -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'''
2026-03-14 09:44:27.427999 | controller |     - '    logger.go:42: 09:40:54 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.428005 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.428011 | controller |     - '    logger.go:42: 09:40:54 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.428017 | controller |       ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.428022 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.428028 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.428034 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_test:8df261d072f382ca7aeb59823b1f461f@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.428040 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.428046 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.428051 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.428057 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.428063 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.428068 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.428074 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.428080 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.428085 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.428091 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.428097 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.428103 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.428109 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.428114 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.428120 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.428125 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.428131 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.428136 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.428145 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.428150 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]''
2026-03-14 09:44:27.428156 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.428162 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.428167 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.428173 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.428179 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.428199 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.428206 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.428213 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.428218 | controller |     - '    logger.go:42: 09:40:54 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.428224 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.428230 | controller |     - '    logger.go:42: 09:40:54 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.428236 | controller |       -czPo ''TimeOut 80'''
2026-03-14 09:44:27.428242 | controller |     - '    logger.go:42: 09:40:54 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.428248 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf'
2026-03-14 09:44:27.428254 | controller |     - '    logger.go:42: 09:40:54 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.428259 | controller |       ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:44:27.428265 | controller |       ''*:9322>'' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:44:27.428271 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.428277 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.428282 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.428302 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:44:27.428311 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:44:27.428316 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:44:27.428322 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:44:27.428328 | controller |       internal display-name=internal group=watcher processes=2 threads=1 user=watcher
2026-03-14 09:44:27.428334 | controller |       WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>''
2026-03-14 09:44:27.428341 | controller |       ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:44:27.428346 | controller |       ''*:9322>'' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:44:27.428352 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.428358 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.428364 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.428370 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:44:27.428381 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:44:27.428387 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:44:27.428393 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:44:27.428399 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:44:27.428404 | controller |       public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>'''
2026-03-14 09:44:27.428410 | controller |     - '    logger.go:42: 09:40:54 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.428416 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.428421 | controller |     - '    logger.go:42: 09:40:55 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.428427 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.428432 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.428438 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.428444 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.428449 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.428455 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.428461 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.428467 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.428473 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.428479 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.428485 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.428491 | controller |       == 1 ]'
2026-03-14 09:44:27.428496 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.428502 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.428508 | controller |     - '        else'
2026-03-14 09:44:27.428514 | controller |     - '          exit 1'
2026-03-14 09:44:27.428530 | controller |     - '        fi'
2026-03-14 09:44:27.428536 | controller |     - '        ]'
2026-03-14 09:44:27.428542 | controller |     - '    logger.go:42: 09:40:55 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.428548 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.428554 | controller |     - '    logger.go:42: 09:40:55 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.428559 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.428565 | controller |     - '    logger.go:42: 09:40:55 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.428571 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.428577 | controller |     - '    logger.go:42: 09:40:55 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.428583 | controller |       -1'
2026-03-14 09:44:27.428597 | controller |     - '    logger.go:42: 09:40:55 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.428603 | controller |       -v ''^$'''
2026-03-14 09:44:27.428609 | controller |     - '    logger.go:42: 09:40:55 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.428615 | controller |     - '    logger.go:42: 09:40:55 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.428620 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.428628 | controller |     - '    logger.go:42: 09:40:55 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.428634 | controller |       -c ''^# Global config'''
2026-03-14 09:44:27.428640 | controller |     - '    logger.go:42: 09:40:55 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.428649 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:44:27.428655 | controller |     - '    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.428661 | controller |       ''#'' Global config'
2026-03-14 09:44:27.428667 | controller |     - '    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.428672 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.428678 | controller |     - '    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.428684 | controller |       -c ''^# Service config'''
2026-03-14 09:44:27.428690 | controller |     - '    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.428696 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf'
2026-03-14 09:44:27.428702 | controller |     - '    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.428708 | controller |       ''#'' Service config'
2026-03-14 09:44:27.428714 | controller |     - '    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.428720 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.428726 | controller |     - '    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.428732 | controller |       -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'''
2026-03-14 09:44:27.428738 | controller |     - '    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.428744 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.428749 | controller |     - '    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.428755 | controller |       ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.428761 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.428770 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.428776 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_test:8df261d072f382ca7aeb59823b1f461f@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.428782 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.428788 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.428794 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.428799 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.428805 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.428811 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.428817 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.428823 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.428829 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.428835 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.428849 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.428856 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.428865 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.428871 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.428877 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.428883 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.428889 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.428894 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.428900 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.428907 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]''
2026-03-14 09:44:27.428913 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.428920 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.428927 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.428933 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.428941 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.428947 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.428953 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.428959 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.428965 | controller |     - '    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.428971 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.428980 | controller |     - '    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.428986 | controller |       -czPo ''TimeOut 80'''
2026-03-14 09:44:27.428992 | controller |     - '    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.428998 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf'
2026-03-14 09:44:27.429004 | controller |     - '    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.429010 | controller |       ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:44:27.429016 | controller |       ''*:9322>'' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:44:27.429022 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.429028 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.429034 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.429040 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:44:27.429046 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:44:27.429052 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:44:27.429058 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:44:27.429064 | controller |       internal display-name=internal group=watcher processes=2 threads=1 user=watcher
2026-03-14 09:44:27.429070 | controller |       WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>''
2026-03-14 09:44:27.429079 | controller |       ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:44:27.429085 | controller |       ''*:9322>'' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:44:27.429090 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.429096 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.429102 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.429108 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:44:27.429114 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:44:27.429120 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:44:27.429125 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:44:27.429140 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:44:27.429147 | controller |       public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>'''
2026-03-14 09:44:27.429153 | controller |     - '    logger.go:42: 09:40:56 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.429159 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.429165 | controller |     - '    logger.go:42: 09:40:57 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.429171 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.429177 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.429183 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.429193 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.429199 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.429205 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.429211 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.429217 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.429223 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.429228 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.429234 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.429240 | controller |       == 1 ]'
2026-03-14 09:44:27.429246 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.429252 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.429258 | controller |     - '        else'
2026-03-14 09:44:27.429264 | controller |     - '          exit 1'
2026-03-14 09:44:27.429270 | controller |     - '        fi'
2026-03-14 09:44:27.429276 | controller |     - '        ]'
2026-03-14 09:44:27.429282 | controller |     - '    logger.go:42: 09:40:57 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.429302 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.429312 | controller |     - '    logger.go:42: 09:40:57 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.429318 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.429324 | controller |     - '    logger.go:42: 09:40:57 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.429330 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.429336 | controller |     - '    logger.go:42: 09:40:57 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.429342 | controller |       -1'
2026-03-14 09:44:27.429350 | controller |     - '    logger.go:42: 09:40:57 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.429356 | controller |       -v ''^$'''
2026-03-14 09:44:27.429362 | controller |     - '    logger.go:42: 09:40:57 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.429368 | controller |     - '    logger.go:42: 09:40:57 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.429374 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.429383 | controller |     - '    logger.go:42: 09:40:57 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.429388 | controller |       -c ''^# Global config'''
2026-03-14 09:44:27.429394 | controller |     - '    logger.go:42: 09:40:57 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.429400 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:44:27.429405 | controller |     - '    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.429411 | controller |       ''#'' Global config'
2026-03-14 09:44:27.429416 | controller |     - '    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.429422 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.429428 | controller |     - '    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.429434 | controller |       -c ''^# Service config'''
2026-03-14 09:44:27.429440 | controller |     - '    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.429446 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf'
2026-03-14 09:44:27.429452 | controller |     - '    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.429457 | controller |       ''#'' Service config'
2026-03-14 09:44:27.429463 | controller |     - '    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.429469 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.429475 | controller |     - '    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.429483 | controller |       -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'''
2026-03-14 09:44:27.429489 | controller |     - '    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.429495 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.429500 | controller |     - '    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.429515 | controller |       ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.429522 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.429528 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.429534 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_test:8df261d072f382ca7aeb59823b1f461f@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.429540 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.429546 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.429552 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.429558 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.429563 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.429569 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.429578 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.429584 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.429589 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.429595 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.429601 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.429607 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.429613 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.429618 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.429624 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.429630 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.429638 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.429645 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.429650 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.429656 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]''
2026-03-14 09:44:27.429662 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.429668 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.429674 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.429679 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.429685 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.429691 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.429697 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.429703 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.429708 | controller |     - '    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.429714 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.429720 | controller |     - '    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.429726 | controller |       -czPo ''TimeOut 80'''
2026-03-14 09:44:27.429731 | controller |     - '    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.429739 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf'
2026-03-14 09:44:27.429744 | controller |     - '    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.429748 | controller |       ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:44:27.429752 | controller |       ''*:9322>'' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:44:27.429756 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.429761 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.429772 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.429779 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:44:27.429785 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:44:27.429791 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:44:27.429796 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:44:27.429802 | controller |       internal display-name=internal group=watcher processes=2 threads=1 user=watcher
2026-03-14 09:44:27.429807 | controller |       WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>''
2026-03-14 09:44:27.429813 | controller |       ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:44:27.429818 | controller |       ''*:9322>'' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:44:27.429823 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.429829 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.429834 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.429840 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:44:27.429846 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:44:27.429851 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:44:27.429857 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:44:27.429862 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:44:27.429868 | controller |       public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>'''
2026-03-14 09:44:27.429873 | controller |     - '    logger.go:42: 09:40:58 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.429879 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.429885 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.429891 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.429897 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.429902 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.429908 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.429914 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.429919 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.429925 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.429931 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.429936 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.429942 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.429948 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.429953 | controller |       == 1 ]'
2026-03-14 09:44:27.429959 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.429965 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.429970 | controller |     - '        else'
2026-03-14 09:44:27.429976 | controller |     - '          exit 1'
2026-03-14 09:44:27.429982 | controller |     - '        fi'
2026-03-14 09:44:27.429987 | controller |     - '        ]'
2026-03-14 09:44:27.429993 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.430002 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.430008 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.430013 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.430019 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.430025 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.430031 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.430036 | controller |       -1'
2026-03-14 09:44:27.430042 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.430048 | controller |       -v ''^$'''
2026-03-14 09:44:27.430054 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.430060 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.430066 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.430072 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.430077 | controller |       -c ''^# Global config'''
2026-03-14 09:44:27.430083 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.430101 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:44:27.430109 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.430115 | controller |       ''#'' Global config'
2026-03-14 09:44:27.430120 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.430126 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.430132 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.430138 | controller |       -c ''^# Service config'''
2026-03-14 09:44:27.430144 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.430150 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf'
2026-03-14 09:44:27.430155 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.430161 | controller |       ''#'' Service config'
2026-03-14 09:44:27.430167 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.430173 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.430179 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.430184 | controller |       -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'''
2026-03-14 09:44:27.430270 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.430277 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.430283 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.430289 | controller |       ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.430315 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.430320 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.430324 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_test:8df261d072f382ca7aeb59823b1f461f@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.430331 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.430335 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.430339 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.430344 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.430348 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.430352 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.430356 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.430360 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.430365 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.430369 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.430373 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.430377 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.430382 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.430386 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.430390 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.430394 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.430398 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.430403 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.430407 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.430411 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]''
2026-03-14 09:44:27.430415 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.430419 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.430431 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.430436 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.430440 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.430444 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.430448 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.430452 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.430457 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.430461 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.430465 | controller |     - '    logger.go:42: 09:41:00 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.430469 | controller |       -czPo ''TimeOut 80'''
2026-03-14 09:44:27.430473 | controller |     - '    logger.go:42: 09:41:01 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.430477 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf'
2026-03-14 09:44:27.430484 | controller |     - '    logger.go:42: 09:41:01 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.430488 | controller |       ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:44:27.430494 | controller |       ''*:9322>'' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:44:27.430499 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.430503 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.430507 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.430511 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:44:27.430516 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:44:27.430520 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:44:27.430524 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:44:27.430528 | controller |       internal display-name=internal group=watcher processes=2 threads=1 user=watcher
2026-03-14 09:44:27.430534 | controller |       WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>''
2026-03-14 09:44:27.430539 | controller |       ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:44:27.430543 | controller |       ''*:9322>'' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:44:27.430547 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.430552 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.430556 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.430560 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:44:27.430564 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:44:27.430569 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:44:27.430573 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:44:27.430577 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:44:27.430581 | controller |       public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>'''
2026-03-14 09:44:27.430586 | controller |     - '    logger.go:42: 09:41:01 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.430590 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.430594 | controller |     - '    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.430598 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.430603 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.430607 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.430611 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.430615 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.430620 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.430624 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.430628 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.430632 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.430637 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.430646 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.430653 | controller |       == 1 ]'
2026-03-14 09:44:27.430658 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.430662 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.430667 | controller |     - '        else'
2026-03-14 09:44:27.430671 | controller |     - '          exit 1'
2026-03-14 09:44:27.430675 | controller |     - '        fi'
2026-03-14 09:44:27.430679 | controller |     - '        ]'
2026-03-14 09:44:27.430684 | controller |     - '    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.430688 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.430692 | controller |     - '    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.430697 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.430701 | controller |     - '    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.430705 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.430709 | controller |     - '    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.430714 | controller |       -1'
2026-03-14 09:44:27.430718 | controller |     - '    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.430722 | controller |       -v ''^$'''
2026-03-14 09:44:27.430727 | controller |     - '    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.430731 | controller |     - '    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.430735 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.430739 | controller |     - '    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.430744 | controller |       -c ''^# Global config'''
2026-03-14 09:44:27.430751 | controller |     - '    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.430756 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:44:27.430760 | controller |     - '    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.430765 | controller |       ''#'' Global config'
2026-03-14 09:44:27.430771 | controller |     - '    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.430777 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.430785 | controller |     - '    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.430791 | controller |       -c ''^# Service config'''
2026-03-14 09:44:27.430797 | controller |     - '    logger.go:42: 09:41:02 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.430803 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf'
2026-03-14 09:44:27.430809 | controller |     - '    logger.go:42: 09:41:03 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.430815 | controller |       ''#'' Service config'
2026-03-14 09:44:27.430821 | controller |     - '    logger.go:42: 09:41:03 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.430827 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.430833 | controller |     - '    logger.go:42: 09:41:03 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.430838 | controller |       -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'''
2026-03-14 09:44:27.430844 | controller |     - '    logger.go:42: 09:41:03 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.430850 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.430856 | controller |     - '    logger.go:42: 09:41:03 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.430864 | controller |       ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.430870 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.430876 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.430882 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_test:8df261d072f382ca7aeb59823b1f461f@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.430888 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.430894 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.430900 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.430906 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.430912 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.430918 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.430924 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.430940 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.430947 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.430953 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.430960 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.430966 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.430972 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.430978 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.430984 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.430989 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.430995 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.431001 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.431006 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.431012 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]''
2026-03-14 09:44:27.431018 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.431024 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.431030 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.431035 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.431041 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.431046 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.431052 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.431060 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.431066 | controller |     - '    logger.go:42: 09:41:03 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.431072 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.431078 | controller |     - '    logger.go:42: 09:41:03 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.431083 | controller |       -czPo ''TimeOut 80'''
2026-03-14 09:44:27.431089 | controller |     - '    logger.go:42: 09:41:03 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.431095 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf'
2026-03-14 09:44:27.431101 | controller |     - '    logger.go:42: 09:41:03 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.431106 | controller |       ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:44:27.431112 | controller |       ''*:9322>'' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:44:27.431117 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.431126 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.431132 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.431137 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:44:27.431143 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:44:27.431148 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:44:27.431153 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:44:27.431159 | controller |       internal display-name=internal group=watcher processes=2 threads=1 user=watcher
2026-03-14 09:44:27.431164 | controller |       WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>''
2026-03-14 09:44:27.431169 | controller |       ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:44:27.431175 | controller |       ''*:9322>'' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:44:27.431180 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.431186 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.431191 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.431196 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:44:27.431210 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:44:27.431217 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:44:27.431222 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:44:27.431227 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:44:27.431233 | controller |       public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>'''
2026-03-14 09:44:27.431238 | controller |     - '    logger.go:42: 09:41:03 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.431243 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.431249 | controller |     - '    logger.go:42: 09:41:04 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.431254 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.431260 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.431266 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.431272 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.431280 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.431286 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.431313 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.431320 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.431326 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.431331 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.431337 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.431343 | controller |       == 1 ]'
2026-03-14 09:44:27.431349 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.431355 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.431361 | controller |     - '        else'
2026-03-14 09:44:27.431367 | controller |     - '          exit 1'
2026-03-14 09:44:27.431372 | controller |     - '        fi'
2026-03-14 09:44:27.431378 | controller |     - '        ]'
2026-03-14 09:44:27.431384 | controller |     - '    logger.go:42: 09:41:04 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.431390 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.431396 | controller |     - '    logger.go:42: 09:41:04 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.431402 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.431408 | controller |     - '    logger.go:42: 09:41:04 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.431414 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.431420 | controller |     - '    logger.go:42: 09:41:04 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.431425 | controller |       -1'
2026-03-14 09:44:27.431431 | controller |     - '    logger.go:42: 09:41:04 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.431437 | controller |       -v ''^$'''
2026-03-14 09:44:27.431447 | controller |     - '    logger.go:42: 09:41:04 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.431451 | controller |     - '    logger.go:42: 09:41:04 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.431456 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.431460 | controller |     - '    logger.go:42: 09:41:04 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.431464 | controller |       -c ''^# Global config'''
2026-03-14 09:44:27.431469 | controller |     - '    logger.go:42: 09:41:04 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.431473 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:44:27.431477 | controller |     - '    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.431482 | controller |       ''#'' Global config'
2026-03-14 09:44:27.431486 | controller |     - '    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.431490 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.431494 | controller |     - '    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.431500 | controller |       -c ''^# Service config'''
2026-03-14 09:44:27.431504 | controller |     - '    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.431508 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf'
2026-03-14 09:44:27.431515 | controller |     - '    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.431521 | controller |       ''#'' Service config'
2026-03-14 09:44:27.431526 | controller |     - '    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.431535 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.431541 | controller |     - '    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.431546 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.431552 | controller |     - '    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.431567 | controller |       -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'''
2026-03-14 09:44:27.431573 | controller |     - '    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.431578 | controller |       ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.431582 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.431586 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.431590 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_test:8df261d072f382ca7aeb59823b1f461f@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.431595 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.431599 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.431603 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.431607 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.431611 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.431616 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.431620 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.431624 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.431628 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.431632 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.431637 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.431641 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.431645 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.431649 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.431653 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.431657 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.431662 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.431666 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.431670 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.431674 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]''
2026-03-14 09:44:27.431678 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.431683 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.431689 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.431693 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.431699 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.431704 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.431710 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.431716 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.431721 | controller |     - '    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.431727 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.431733 | controller |     - '    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.431739 | controller |       -czPo ''TimeOut 80'''
2026-03-14 09:44:27.431745 | controller |     - '    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.431751 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf'
2026-03-14 09:44:27.431757 | controller |     - '    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.431762 | controller |       ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:44:27.431773 | controller |       ''*:9322>'' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:44:27.431777 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.431782 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.431786 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.431793 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:44:27.431797 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:44:27.431801 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:44:27.431805 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:44:27.431810 | controller |       internal display-name=internal group=watcher processes=2 threads=1 user=watcher
2026-03-14 09:44:27.431814 | controller |       WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>''
2026-03-14 09:44:27.431818 | controller |       ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:44:27.431822 | controller |       ''*:9322>'' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:44:27.431826 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.431831 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.431835 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.431839 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:44:27.431843 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:44:27.431847 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:44:27.431851 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:44:27.431856 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:44:27.431862 | controller |       public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>'''
2026-03-14 09:44:27.431866 | controller |     - '    logger.go:42: 09:41:05 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.431870 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.431874 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | running
2026-03-14 09:44:27.431878 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.431883 | controller |     - '        oc project watcher-kuttl-default'
2026-03-14 09:44:27.431887 | controller |     - '        APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api"
2026-03-14 09:44:27.431891 | controller |       -ocustom-columns=:metadata.name|grep -v ^$|head -1)'
2026-03-14 09:44:27.431895 | controller |     - '        if [ -n "${APIPOD}" ]; then'
2026-03-14 09:44:27.431899 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf)
2026-03-14 09:44:27.431904 | controller |       |grep -c "^# Global config") == 1 ]'
2026-03-14 09:44:27.431908 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf)
2026-03-14 09:44:27.431912 | controller |       |grep -c "^# Service config") == 1 ]'
2026-03-14 09:44:27.431916 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf)
2026-03-14 09:44:27.431921 | controller |       |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'')
2026-03-14 09:44:27.431925 | controller |       == 1 ]'
2026-03-14 09:44:27.431929 | controller |     - '          [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf)
2026-03-14 09:44:27.431933 | controller |       |grep -czPo ''TimeOut 80'') == 1 ]'
2026-03-14 09:44:27.431937 | controller |     - '        else'
2026-03-14 09:44:27.431942 | controller |     - '          exit 1'
2026-03-14 09:44:27.431946 | controller |     - '        fi'
2026-03-14 09:44:27.431950 | controller |     - '        ]'
2026-03-14 09:44:27.431954 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | + oc project
2026-03-14 09:44:27.431959 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.431963 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | Already
2026-03-14 09:44:27.431967 | controller |       on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".'
2026-03-14 09:44:27.431971 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | ++ oc get
2026-03-14 09:44:27.431975 | controller |       pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name'
2026-03-14 09:44:27.431979 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | ++ head
2026-03-14 09:44:27.431984 | controller |       -1'
2026-03-14 09:44:27.431988 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.431992 | controller |       -v ''^$'''
2026-03-14 09:44:27.431996 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0'
2026-03-14 09:44:27.432000 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.432010 | controller |       -n watcher-kuttl-api-0 '']'''
2026-03-14 09:44:27.432014 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.432018 | controller |       -c ''^# Global config'''
2026-03-14 09:44:27.432022 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.432027 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf'
2026-03-14 09:44:27.432031 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.432035 | controller |       ''#'' Global config'
2026-03-14 09:44:27.432039 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.432043 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.432049 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.432053 | controller |       -c ''^# Service config'''
2026-03-14 09:44:27.432057 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.432062 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf'
2026-03-14 09:44:27.432066 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.432071 | controller |       ''#'' Service config'
2026-03-14 09:44:27.432076 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.432081 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.432088 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.432093 | controller |       -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'''
2026-03-14 09:44:27.432098 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.432103 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf'
2026-03-14 09:44:27.432108 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.432113 | controller |       ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_N28IdurgXrXRyjcDjm5:SWlAUOltabpX60RHkCVAFu92TV5cnoTx@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1''
2026-03-14 09:44:27.432118 | controller |       control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log
2026-03-14 09:44:27.432123 | controller |       ''#'' empty notification_level means that no notification will be sent notification_level
2026-03-14 09:44:27.432128 | controller |       = ''[database]'' connection = ''mysql+pymysql://watcher_test:8df261d072f382ca7aeb59823b1f461f@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf''
2026-03-14 09:44:27.432133 | controller |       ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]''
2026-03-14 09:44:27.432254 | controller |       driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true
2026-03-14 09:44:27.432259 | controller |       amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.432263 | controller |       tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile
2026-03-14 09:44:27.432267 | controller |       = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt
2026-03-14 09:44:27.432271 | controller |       memcache_tls_enabled = true project_domain_name = Default project_name = service
2026-03-14 09:44:27.432276 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.432281 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.432286 | controller |       internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
2026-03-14 09:44:27.432311 | controller |       ''[watcher_clients_auth]'' project_domain_name = Default project_name = service
2026-03-14 09:44:27.432317 | controller |       user_domain_name = Default password = password username = watcher auth_type = password
2026-03-14 09:44:27.432323 | controller |       auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface =
2026-03-14 09:44:27.432329 | controller |       internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]''
2026-03-14 09:44:27.432335 | controller |       lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus
2026-03-14 09:44:27.432341 | controller |       ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212
2026-03-14 09:44:27.432346 | controller |       memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true
2026-03-14 09:44:27.432355 | controller |       tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key
2026-03-14 09:44:27.432360 | controller |       tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]''
2026-03-14 09:44:27.432365 | controller |       host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile =
2026-03-14 09:44:27.432377 | controller |       /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]''
2026-03-14 09:44:27.432381 | controller |       endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type
2026-03-14 09:44:27.432385 | controller |       = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal
2026-03-14 09:44:27.432390 | controller |       region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name
2026-03-14 09:44:27.432396 | controller |       = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]''
2026-03-14 09:44:27.432402 | controller |       endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type
2026-03-14 09:44:27.432408 | controller |       = internal region_name = regionOne ''[placement_client]'' interface = internal region_name
2026-03-14 09:44:27.432414 | controller |       = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]''
2026-03-14 09:44:27.432420 | controller |       period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900'
2026-03-14 09:44:27.432426 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.432431 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.432480 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | ++ grep
2026-03-14 09:44:27.432491 | controller |       -czPo ''TimeOut 80'''
2026-03-14 09:44:27.432497 | controller |     - '    logger.go:42: 09:41:07 | watcher/4-deploy-with-precreated-account | +++ oc
2026-03-14 09:44:27.432503 | controller |       rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf'
2026-03-14 09:44:27.432508 | controller |     - '    logger.go:42: 09:41:08 | watcher/4-deploy-with-precreated-account | ++ echo
2026-03-14 09:44:27.432514 | controller |       ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:44:27.432520 | controller |       ''*:9322>'' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:44:27.432525 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.432531 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.432536 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.432542 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:44:27.432549 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:44:27.432558 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:44:27.432564 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:44:27.432570 | controller |       internal display-name=internal group=watcher processes=2 threads=1 user=watcher
2026-03-14 09:44:27.432576 | controller |       WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>''
2026-03-14 09:44:27.432582 | controller |       ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration ''<VirtualHost''
2026-03-14 09:44:27.432587 | controller |       ''*:9322>'' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot
2026-03-14 09:44:27.432593 | controller |       DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut
2026-03-14 09:44:27.432598 | controller |       80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin
2026-03-14 09:44:27.432607 | controller |       ''<Directory'' ''"/var/www/cgi-bin">'' Options -Indexes +FollowSymLinks +MultiViews
2026-03-14 09:44:27.432613 | controller |       AllowOverride None Require all granted ''</Directory>'' ''##'' Logging ErrorLog
2026-03-14 09:44:27.432619 | controller |       /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded''
2026-03-14 09:44:27.432624 | controller |       CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug
2026-03-14 09:44:27.432630 | controller |       LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess
2026-03-14 09:44:27.432636 | controller |       public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup
2026-03-14 09:44:27.432642 | controller |       public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''</VirtualHost>'''
2026-03-14 09:44:27.432648 | controller |     - '    logger.go:42: 09:41:08 | watcher/4-deploy-with-precreated-account | + ''[''
2026-03-14 09:44:27.432654 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.432660 | controller |     - '    logger.go:42: 09:41:08 | watcher/4-deploy-with-precreated-account | test step
2026-03-14 09:44:27.432666 | controller |       completed 4-deploy-with-precreated-account'
2026-03-14 09:44:27.432671 | controller |     - '    logger.go:42: 09:41:08 | watcher/5-cleanup-watcher | starting test step 5-cleanup-watcher'
2026-03-14 09:44:27.432677 | controller |     - '    logger.go:42: 09:41:08 | watcher/5-cleanup-watcher | running command: [sh -c
2026-03-14 09:44:27.432683 | controller |       set -ex'
2026-03-14 09:44:27.432689 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.432695 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:44:27.432701 | controller |     - '        ]'
2026-03-14 09:44:27.432706 | controller |     - '    logger.go:42: 09:41:08 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.432722 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.432730 | controller |     - '    logger.go:42: 09:41:08 | watcher/5-cleanup-watcher | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.432736 | controller |     - '    logger.go:42: 09:41:11 | watcher/5-cleanup-watcher | + ''['' 1 == 0 '']'''
2026-03-14 09:44:27.432742 | controller |     - '    logger.go:42: 09:41:13 | watcher/5-cleanup-watcher | running command: [sh -c
2026-03-14 09:44:27.432748 | controller |       set -ex'
2026-03-14 09:44:27.432754 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.432761 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:44:27.432767 | controller |     - '        ]'
2026-03-14 09:44:27.432773 | controller |     - '    logger.go:42: 09:41:13 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.432779 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.432785 | controller |     - '    logger.go:42: 09:41:13 | watcher/5-cleanup-watcher | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.432791 | controller |     - '    logger.go:42: 09:41:15 | watcher/5-cleanup-watcher | + ''['' 0 == 0 '']'''
2026-03-14 09:44:27.432798 | controller |     - '    logger.go:42: 09:41:16 | watcher/5-cleanup-watcher | running command: [sh -c
2026-03-14 09:44:27.432804 | controller |       set -ex'
2026-03-14 09:44:27.432810 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.432815 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:44:27.432821 | controller |     - '        ]'
2026-03-14 09:44:27.432828 | controller |     - '    logger.go:42: 09:41:16 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default
2026-03-14 09:44:27.432833 | controller |       openstackclient -- openstack service list -f value -c Name -c Type'
2026-03-14 09:44:27.432840 | controller |     - '    logger.go:42: 09:41:16 | watcher/5-cleanup-watcher | ++ grep -c ''^watcher'''
2026-03-14 09:44:27.432846 | controller |     - '    logger.go:42: 09:41:18 | watcher/5-cleanup-watcher | + ''['' 0 == 0 '']'''
2026-03-14 09:44:27.432852 | controller |     - '    logger.go:42: 09:41:18 | watcher/5-cleanup-watcher | test step completed 5-cleanup-watcher'
2026-03-14 09:44:27.432861 | controller |     - '    logger.go:42: 09:41:18 | watcher | skipping kubernetes event logging'
2026-03-14 09:44:27.432867 | controller |     - === CONT  kuttl/harness/watcher-cinder
2026-03-14 09:44:27.432873 | controller |     - '    logger.go:42: 09:41:18 | watcher-cinder | Skipping creation of user-supplied
2026-03-14 09:44:27.432879 | controller |       namespace: watcher-kuttl-default'
2026-03-14 09:44:27.432886 | controller |     - '    logger.go:42: 09:41:18 | watcher-cinder/0-cleanup-watcher | starting test step
2026-03-14 09:44:27.432892 | controller |       0-cleanup-watcher'
2026-03-14 09:44:27.432898 | controller |     - '    logger.go:42: 09:41:18 | watcher-cinder/0-cleanup-watcher | test step completed
2026-03-14 09:44:27.432905 | controller |       0-cleanup-watcher'
2026-03-14 09:44:27.432911 | controller |     - '    logger.go:42: 09:41:18 | watcher-cinder/1-deploy-watcher-no-cinder | starting
2026-03-14 09:44:27.432917 | controller |       test step 1-deploy-watcher-no-cinder'
2026-03-14 09:44:27.432924 | controller |     - '    logger.go:42: 09:41:18 | watcher-cinder/1-deploy-watcher-no-cinder | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:44:27.432930 | controller |       created'
2026-03-14 09:44:27.432936 | controller |     - '    logger.go:42: 09:41:18 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.432942 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.432948 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.432954 | controller |       service'
2026-03-14 09:44:27.432963 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.432970 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.432974 | controller |     - '        ]'
2026-03-14 09:44:27.432978 | controller |     - '    logger.go:42: 09:41:18 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.432982 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.432986 | controller |     - '    logger.go:42: 09:41:18 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.432991 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.432995 | controller |     - '    logger.go:42: 09:41:18 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.432999 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:44:27.433003 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:44:27.433007 | controller |     - '    logger.go:42: 09:41:18 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.433012 | controller |       0 == 2 '']'''
2026-03-14 09:44:27.433018 | controller |     - '    logger.go:42: 09:41:19 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.433024 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.433030 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.433036 | controller |       service'
2026-03-14 09:44:27.433042 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.433047 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.433052 | controller |     - '        ]'
2026-03-14 09:44:27.433057 | controller |     - '    logger.go:42: 09:41:19 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.433062 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.433068 | controller |     - '    logger.go:42: 09:41:19 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.433226 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.433232 | controller |     - '    logger.go:42: 09:41:20 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.433238 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:44:27.433244 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:44:27.433250 | controller |     - '    logger.go:42: 09:41:20 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.433258 | controller |       0 == 2 '']'''
2026-03-14 09:44:27.433264 | controller |     - '    logger.go:42: 09:41:21 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.433269 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.433274 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.433278 | controller |       service'
2026-03-14 09:44:27.433282 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.433286 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.433317 | controller |     - '        ]'
2026-03-14 09:44:27.433323 | controller |     - '    logger.go:42: 09:41:21 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.433329 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.433335 | controller |     - '    logger.go:42: 09:41:21 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.433341 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.433347 | controller |     - '    logger.go:42: 09:41:21 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.433353 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:44:27.433358 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:44:27.433364 | controller |     - '    logger.go:42: 09:41:21 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.433370 | controller |       0 == 2 '']'''
2026-03-14 09:44:27.433376 | controller |     - '    logger.go:42: 09:41:22 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.433381 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.433387 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.433393 | controller |       service'
2026-03-14 09:44:27.433399 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.433405 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.433410 | controller |     - '        ]'
2026-03-14 09:44:27.433416 | controller |     - '    logger.go:42: 09:41:22 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.433421 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.433425 | controller |     - '    logger.go:42: 09:41:22 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.433430 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.433434 | controller |     - '    logger.go:42: 09:41:22 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.433438 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:44:27.433442 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:44:27.433446 | controller |     - '    logger.go:42: 09:41:22 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.433451 | controller |       0 == 2 '']'''
2026-03-14 09:44:27.433455 | controller |     - '    logger.go:42: 09:41:23 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.433459 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.433467 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.433471 | controller |       service'
2026-03-14 09:44:27.433475 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.433479 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.433483 | controller |     - '        ]'
2026-03-14 09:44:27.433488 | controller |     - '    logger.go:42: 09:41:23 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.433492 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.433496 | controller |     - '    logger.go:42: 09:41:23 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.433502 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.433506 | controller |     - '    logger.go:42: 09:41:23 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.433510 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:44:27.433515 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:44:27.433519 | controller |     - '    logger.go:42: 09:41:23 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.433523 | controller |       0 == 2 '']'''
2026-03-14 09:44:27.433527 | controller |     - '    logger.go:42: 09:41:24 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.433532 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.433537 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.433542 | controller |       service'
2026-03-14 09:44:27.433546 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.433550 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.433555 | controller |     - '        ]'
2026-03-14 09:44:27.433559 | controller |     - '    logger.go:42: 09:41:24 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.433571 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.433576 | controller |     - '    logger.go:42: 09:41:24 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.433580 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.433584 | controller |     - '    logger.go:42: 09:41:24 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.433589 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:44:27.433593 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:44:27.433597 | controller |     - '    logger.go:42: 09:41:24 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.433602 | controller |       0 == 2 '']'''
2026-03-14 09:44:27.433606 | controller |     - '    logger.go:42: 09:41:25 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.433610 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.433614 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.433619 | controller |       service'
2026-03-14 09:44:27.433623 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.433627 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.433631 | controller |     - '        ]'
2026-03-14 09:44:27.433635 | controller |     - '    logger.go:42: 09:41:25 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.433640 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.433644 | controller |     - '    logger.go:42: 09:41:25 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.433648 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.433652 | controller |     - '    logger.go:42: 09:41:25 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.433656 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:44:27.433661 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:44:27.433665 | controller |     - '    logger.go:42: 09:41:25 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.433669 | controller |       0 == 2 '']'''
2026-03-14 09:44:27.433673 | controller |     - '    logger.go:42: 09:41:26 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.433677 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.433682 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.433686 | controller |       service'
2026-03-14 09:44:27.433690 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.433696 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.433700 | controller |     - '        ]'
2026-03-14 09:44:27.433704 | controller |     - '    logger.go:42: 09:41:26 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.433709 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.433713 | controller |     - '    logger.go:42: 09:41:26 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.433717 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.433721 | controller |     - '    logger.go:42: 09:41:27 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.433725 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:44:27.433729 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:44:27.433734 | controller |     - '    logger.go:42: 09:41:27 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.433738 | controller |       0 == 2 '']'''
2026-03-14 09:44:27.433744 | controller |     - '    logger.go:42: 09:41:28 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.433748 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.433752 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.433756 | controller |       service'
2026-03-14 09:44:27.433760 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.433765 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.433769 | controller |     - '        ]'
2026-03-14 09:44:27.433773 | controller |     - '    logger.go:42: 09:41:28 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.433779 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.433783 | controller |     - '    logger.go:42: 09:41:28 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.433787 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.433791 | controller |     - '    logger.go:42: 09:41:28 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.433795 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:44:27.433799 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:44:27.433804 | controller |     - '    logger.go:42: 09:41:28 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.433808 | controller |       0 == 2 '']'''
2026-03-14 09:44:27.433812 | controller |     - '    logger.go:42: 09:41:29 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.433816 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.433820 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.433825 | controller |       service'
2026-03-14 09:44:27.433829 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.433839 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.433843 | controller |     - '        ]'
2026-03-14 09:44:27.433847 | controller |     - '    logger.go:42: 09:41:29 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.433852 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.433856 | controller |     - '    logger.go:42: 09:41:29 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.433860 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.433864 | controller |     - '    logger.go:42: 09:41:29 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.433868 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:44:27.433872 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:44:27.433876 | controller |     - '    logger.go:42: 09:41:29 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.433882 | controller |       0 == 2 '']'''
2026-03-14 09:44:27.433886 | controller |     - '    logger.go:42: 09:41:30 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.433891 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.433895 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.433899 | controller |       service'
2026-03-14 09:44:27.433903 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.433907 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.433912 | controller |     - '        ]'
2026-03-14 09:44:27.433916 | controller |     - '    logger.go:42: 09:41:30 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.433921 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.433926 | controller |     - '    logger.go:42: 09:41:30 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.433931 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.433935 | controller |     - '    logger.go:42: 09:41:30 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.433940 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:44:27.433945 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:44:27.433950 | controller |     - '    logger.go:42: 09:41:30 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.433954 | controller |       0 == 2 '']'''
2026-03-14 09:44:27.433958 | controller |     - '    logger.go:42: 09:41:31 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.433963 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.433967 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.433971 | controller |       service'
2026-03-14 09:44:27.433975 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.433979 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.433983 | controller |     - '        ]'
2026-03-14 09:44:27.433989 | controller |     - '    logger.go:42: 09:41:31 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.433994 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.433998 | controller |     - '    logger.go:42: 09:41:31 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.434002 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.434006 | controller |     - '    logger.go:42: 09:41:31 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.434010 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:44:27.434014 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:44:27.434019 | controller |     - '    logger.go:42: 09:41:31 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.434024 | controller |       0 == 2 '']'''
2026-03-14 09:44:27.434029 | controller |     - '    logger.go:42: 09:41:32 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.434033 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.434037 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.434041 | controller |       service'
2026-03-14 09:44:27.434045 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.434050 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.434054 | controller |     - '        ]'
2026-03-14 09:44:27.434058 | controller |     - '    logger.go:42: 09:41:32 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.434062 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.434066 | controller |     - '    logger.go:42: 09:41:32 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.434070 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.434077 | controller |     - '    logger.go:42: 09:41:32 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.434082 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:44:27.434086 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:44:27.434090 | controller |     - '    logger.go:42: 09:41:32 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.434094 | controller |       0 == 2 '']'''
2026-03-14 09:44:27.434098 | controller |     - '    logger.go:42: 09:41:33 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.434102 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.434112 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.434117 | controller |       service'
2026-03-14 09:44:27.434121 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.434125 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.434129 | controller |     - '        ]'
2026-03-14 09:44:27.434134 | controller |     - '    logger.go:42: 09:41:33 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.434138 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.434142 | controller |     - '    logger.go:42: 09:41:33 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.434146 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.434150 | controller |     - '    logger.go:42: 09:41:33 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.434155 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:44:27.434159 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:44:27.434165 | controller |     - '    logger.go:42: 09:41:33 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.434171 | controller |       0 == 2 '']'''
2026-03-14 09:44:27.434176 | controller |     - '    logger.go:42: 09:41:34 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.434182 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.434188 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.434194 | controller |       service'
2026-03-14 09:44:27.434215 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.434221 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.434227 | controller |     - '        ]'
2026-03-14 09:44:27.434233 | controller |     - '    logger.go:42: 09:41:34 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.434238 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.434244 | controller |     - '    logger.go:42: 09:41:34 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.434250 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.434255 | controller |     - '    logger.go:42: 09:41:35 | watcher-cinder/1-deploy-watcher-no-cinder | error:
2026-03-14 09:44:27.434261 | controller |       error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in
2026-03-14 09:44:27.434267 | controller |       namespace "watcher-kuttl-default"'
2026-03-14 09:44:27.434273 | controller |     - '    logger.go:42: 09:41:35 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.434278 | controller |       0 == 2 '']'''
2026-03-14 09:44:27.434284 | controller |     - '    logger.go:42: 09:41:36 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.434306 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.434317 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.434323 | controller |       service'
2026-03-14 09:44:27.434329 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.434341 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.434347 | controller |     - '        ]'
2026-03-14 09:44:27.434353 | controller |     - '    logger.go:42: 09:41:36 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.434359 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.434365 | controller |     - '    logger.go:42: 09:41:36 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.434371 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.434377 | controller |     - '    logger.go:42: 09:41:36 | watcher-cinder/1-deploy-watcher-no-cinder | Error
2026-03-14 09:44:27.434383 | controller |       from server (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0"
2026-03-14 09:44:27.434389 | controller |       is waiting to start: ContainerCreating'
2026-03-14 09:44:27.434399 | controller |     - '    logger.go:42: 09:41:36 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.434405 | controller |       0 == 2 '']'''
2026-03-14 09:44:27.434411 | controller |     - '    logger.go:42: 09:41:37 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.434417 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.434423 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.434428 | controller |       service'
2026-03-14 09:44:27.434434 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.434440 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.434446 | controller |     - '        ]'
2026-03-14 09:44:27.434451 | controller |     - '    logger.go:42: 09:41:37 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.434457 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.434463 | controller |     - '    logger.go:42: 09:41:37 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.434469 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.434475 | controller |     - '    logger.go:42: 09:41:37 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.434481 | controller |       0 == 2 '']'''
2026-03-14 09:44:27.434487 | controller |     - '    logger.go:42: 09:41:38 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.434493 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.434499 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.434515 | controller |       service'
2026-03-14 09:44:27.434521 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.434527 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.434533 | controller |     - '        ]'
2026-03-14 09:44:27.434538 | controller |     - '    logger.go:42: 09:41:38 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.434544 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.434554 | controller |     - '    logger.go:42: 09:41:38 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.434560 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.434565 | controller |     - '    logger.go:42: 09:41:38 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.434572 | controller |       0 == 2 '']'''
2026-03-14 09:44:27.434578 | controller |     - '    logger.go:42: 09:41:39 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.434583 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.434589 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.434595 | controller |       service'
2026-03-14 09:44:27.434601 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.434607 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.434616 | controller |     - '        ]'
2026-03-14 09:44:27.434622 | controller |     - '    logger.go:42: 09:41:39 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.434628 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.434634 | controller |     - '    logger.go:42: 09:41:39 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.434640 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.434646 | controller |     - '    logger.go:42: 09:41:40 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.434652 | controller |       1 == 2 '']'''
2026-03-14 09:44:27.434658 | controller |     - '    logger.go:42: 09:41:41 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.434664 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.434670 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.434676 | controller |       service'
2026-03-14 09:44:27.434682 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.434688 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.434694 | controller |     - '        ]'
2026-03-14 09:44:27.434700 | controller |     - '    logger.go:42: 09:41:41 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.434705 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.434711 | controller |     - '    logger.go:42: 09:41:41 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.434717 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.434723 | controller |     - '    logger.go:42: 09:41:41 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.434728 | controller |       2 == 2 '']'''
2026-03-14 09:44:27.434734 | controller |     - '    logger.go:42: 09:41:42 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.434740 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.434745 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.434751 | controller |       service'
2026-03-14 09:44:27.434757 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.434762 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.434768 | controller |     - '        ]'
2026-03-14 09:44:27.434774 | controller |     - '    logger.go:42: 09:41:42 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.434784 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.434790 | controller |     - '    logger.go:42: 09:41:42 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.434796 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.434802 | controller |     - '    logger.go:42: 09:41:42 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.434808 | controller |       2 == 2 '']'''
2026-03-14 09:44:27.434816 | controller |     - '    logger.go:42: 09:41:43 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.434822 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.434828 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.434834 | controller |       service'
2026-03-14 09:44:27.434839 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.434845 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.434851 | controller |     - '        ]'
2026-03-14 09:44:27.434857 | controller |     - '    logger.go:42: 09:41:43 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.434863 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.434868 | controller |     - '    logger.go:42: 09:41:43 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.434877 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.434883 | controller |     - '    logger.go:42: 09:41:43 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.434889 | controller |       2 == 2 '']'''
2026-03-14 09:44:27.434906 | controller |     - '    logger.go:42: 09:41:44 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.434914 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.434920 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.434926 | controller |       service'
2026-03-14 09:44:27.434931 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.434937 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.434943 | controller |     - '        ]'
2026-03-14 09:44:27.434949 | controller |     - '    logger.go:42: 09:41:44 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.434956 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.434962 | controller |     - '    logger.go:42: 09:41:44 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.434968 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.434974 | controller |     - '    logger.go:42: 09:41:44 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.434980 | controller |       2 == 2 '']'''
2026-03-14 09:44:27.434986 | controller |     - '    logger.go:42: 09:41:45 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.434991 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.434997 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.435003 | controller |       service'
2026-03-14 09:44:27.435009 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.435015 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.435021 | controller |     - '        ]'
2026-03-14 09:44:27.435027 | controller |     - '    logger.go:42: 09:41:45 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.435033 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.435039 | controller |     - '    logger.go:42: 09:41:45 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.435046 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.435052 | controller |     - '    logger.go:42: 09:41:45 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.435058 | controller |       2 == 2 '']'''
2026-03-14 09:44:27.435065 | controller |     - '    logger.go:42: 09:41:47 | watcher-cinder/1-deploy-watcher-no-cinder | running
2026-03-14 09:44:27.435071 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.435077 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.435084 | controller |       service'
2026-03-14 09:44:27.435090 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block
2026-03-14 09:44:27.435096 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.435102 | controller |     - '        ]'
2026-03-14 09:44:27.435108 | controller |     - '    logger.go:42: 09:41:47 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep
2026-03-14 09:44:27.435115 | controller |       -c ''Block storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.435121 | controller |     - '    logger.go:42: 09:41:47 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc
2026-03-14 09:44:27.435127 | controller |       logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.435134 | controller |     - '    logger.go:42: 09:41:47 | watcher-cinder/1-deploy-watcher-no-cinder | + ''[''
2026-03-14 09:44:27.435140 | controller |       2 == 2 '']'''
2026-03-14 09:44:27.435146 | controller |     - '    logger.go:42: 09:41:47 | watcher-cinder/1-deploy-watcher-no-cinder | test step
2026-03-14 09:44:27.435153 | controller |       completed 1-deploy-watcher-no-cinder'
2026-03-14 09:44:27.435163 | controller |     - '    logger.go:42: 09:41:47 | watcher-cinder/2-deploy-cinder | starting test step
2026-03-14 09:44:27.435168 | controller |       2-deploy-cinder'
2026-03-14 09:44:27.435174 | controller |     - '    logger.go:42: 09:41:47 | watcher-cinder/2-deploy-cinder | OpenStackControlPlane:watcher-kuttl-default/openstack
2026-03-14 09:44:27.435181 | controller |       updated'
2026-03-14 09:44:27.435187 | controller |     - '    logger.go:42: 09:41:47 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.435193 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.435199 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.435205 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.435211 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.435217 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.435223 | controller |     - '        ]'
2026-03-14 09:44:27.435230 | controller |     - '    logger.go:42: 09:41:47 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.435236 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.435242 | controller |     - '    logger.go:42: 09:41:47 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.435249 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.435255 | controller |     - '    logger.go:42: 09:41:47 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.435265 | controller |     - '    logger.go:42: 09:41:48 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.435272 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.435278 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.435283 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.435308 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.435329 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.435337 | controller |     - '        ]'
2026-03-14 09:44:27.435344 | controller |     - '    logger.go:42: 09:41:48 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.435350 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.435357 | controller |     - '    logger.go:42: 09:41:48 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.435363 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.435369 | controller |     - '    logger.go:42: 09:41:48 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.435375 | controller |     - '    logger.go:42: 09:41:49 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.435381 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.435387 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.435394 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.435400 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.435406 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.435413 | controller |     - '        ]'
2026-03-14 09:44:27.435419 | controller |     - '    logger.go:42: 09:41:49 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.435425 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.435432 | controller |     - '    logger.go:42: 09:41:49 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.435438 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.435444 | controller |     - '    logger.go:42: 09:41:49 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.435451 | controller |     - '    logger.go:42: 09:41:50 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.435464 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.435471 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.435477 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.435484 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.435490 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.435496 | controller |     - '        ]'
2026-03-14 09:44:27.435502 | controller |     - '    logger.go:42: 09:41:50 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.435509 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.435515 | controller |     - '    logger.go:42: 09:41:50 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.435522 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.435528 | controller |     - '    logger.go:42: 09:41:50 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.435534 | controller |     - '    logger.go:42: 09:41:51 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.435541 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.435547 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.435554 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.435560 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.435567 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.435573 | controller |     - '        ]'
2026-03-14 09:44:27.435579 | controller |     - '    logger.go:42: 09:41:51 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.435586 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.435592 | controller |     - '    logger.go:42: 09:41:51 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.435598 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.435642 | controller |     - '    logger.go:42: 09:41:52 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.435653 | controller |     - '    logger.go:42: 09:41:53 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.435659 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.435666 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.435672 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.435678 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.435684 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.435689 | controller |     - '        ]'
2026-03-14 09:44:27.435696 | controller |     - '    logger.go:42: 09:41:53 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.435702 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.435707 | controller |     - '    logger.go:42: 09:41:53 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.435713 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.435719 | controller |     - '    logger.go:42: 09:41:53 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.435725 | controller |     - '    logger.go:42: 09:41:54 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.435731 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.435737 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.435752 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.435759 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.435765 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.435775 | controller |     - '        ]'
2026-03-14 09:44:27.435782 | controller |     - '    logger.go:42: 09:41:54 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.435788 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.435794 | controller |     - '    logger.go:42: 09:41:54 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.435800 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.435806 | controller |     - '    logger.go:42: 09:41:54 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.435811 | controller |     - '    logger.go:42: 09:41:55 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.435817 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.435826 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.435852 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.435860 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.435869 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.435875 | controller |     - '        ]'
2026-03-14 09:44:27.435881 | controller |     - '    logger.go:42: 09:41:55 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.435887 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.435893 | controller |     - '    logger.go:42: 09:41:55 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.435899 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.435907 | controller |     - '    logger.go:42: 09:41:55 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.435913 | controller |     - '    logger.go:42: 09:41:56 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.435919 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.435923 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.435927 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.435932 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.435936 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.435940 | controller |     - '        ]'
2026-03-14 09:44:27.435944 | controller |     - '    logger.go:42: 09:41:56 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.435948 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.435953 | controller |     - '    logger.go:42: 09:41:56 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.435957 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.435962 | controller |     - '    logger.go:42: 09:41:56 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.435968 | controller |     - '    logger.go:42: 09:41:57 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.435974 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.435985 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.435991 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.435997 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.436003 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.436009 | controller |     - '        ]'
2026-03-14 09:44:27.436015 | controller |     - '    logger.go:42: 09:41:57 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.436021 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.436026 | controller |     - '    logger.go:42: 09:41:57 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.436032 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.436038 | controller |     - '    logger.go:42: 09:41:58 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.436046 | controller |     - '    logger.go:42: 09:41:59 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.436052 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.436057 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.436063 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.436068 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.436072 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.436077 | controller |     - '        ]'
2026-03-14 09:44:27.436081 | controller |     - '    logger.go:42: 09:41:59 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.436085 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.436089 | controller |     - '    logger.go:42: 09:41:59 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.436094 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.436098 | controller |     - '    logger.go:42: 09:41:59 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.436102 | controller |     - '    logger.go:42: 09:42:00 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.436115 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.436120 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.436124 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.436128 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.436133 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.436137 | controller |     - '        ]'
2026-03-14 09:44:27.436141 | controller |     - '    logger.go:42: 09:42:00 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.436145 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.436149 | controller |     - '    logger.go:42: 09:42:00 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.436154 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.436158 | controller |     - '    logger.go:42: 09:42:00 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.436163 | controller |     - '    logger.go:42: 09:42:01 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.436167 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.436171 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.436175 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.436179 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.436184 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.436188 | controller |     - '        ]'
2026-03-14 09:44:27.436193 | controller |     - '    logger.go:42: 09:42:01 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.436199 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.436205 | controller |     - '    logger.go:42: 09:42:01 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.436211 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.436216 | controller |     - '    logger.go:42: 09:42:01 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.436222 | controller |     - '    logger.go:42: 09:42:02 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.436228 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.436234 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.436240 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.436248 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.436257 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.436261 | controller |     - '        ]'
2026-03-14 09:44:27.436266 | controller |     - '    logger.go:42: 09:42:02 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.436270 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.436274 | controller |     - '    logger.go:42: 09:42:02 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.436278 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.436283 | controller |     - '    logger.go:42: 09:42:02 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.436301 | controller |     - '    logger.go:42: 09:42:03 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.436308 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.436313 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.436317 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.436321 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.436325 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.436330 | controller |     - '        ]'
2026-03-14 09:44:27.436334 | controller |     - '    logger.go:42: 09:42:03 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.436338 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.436342 | controller |     - '    logger.go:42: 09:42:03 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.436346 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.436351 | controller |     - '    logger.go:42: 09:42:03 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.436355 | controller |     - '    logger.go:42: 09:42:05 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.436359 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.436363 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.436367 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.436371 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.436376 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.436380 | controller |     - '        ]'
2026-03-14 09:44:27.436384 | controller |     - '    logger.go:42: 09:42:05 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.436388 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.436392 | controller |     - '    logger.go:42: 09:42:05 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.436397 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.436408 | controller |     - '    logger.go:42: 09:42:05 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.436413 | controller |     - '    logger.go:42: 09:42:06 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.436417 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.436421 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.436425 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.436430 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.436434 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.436438 | controller |     - '        ]'
2026-03-14 09:44:27.436442 | controller |     - '    logger.go:42: 09:42:06 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.436446 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.436451 | controller |     - '    logger.go:42: 09:42:06 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.436457 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.436461 | controller |     - '    logger.go:42: 09:42:06 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.436465 | controller |     - '    logger.go:42: 09:42:07 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.436469 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.436473 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.436478 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.436482 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.436486 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.436490 | controller |     - '        ]'
2026-03-14 09:44:27.436494 | controller |     - '    logger.go:42: 09:42:07 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.436499 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.436503 | controller |     - '    logger.go:42: 09:42:07 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.436507 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.436511 | controller |     - '    logger.go:42: 09:42:07 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.436515 | controller |     - '    logger.go:42: 09:42:08 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.436520 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.436524 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.436528 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.436532 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.436536 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.436541 | controller |     - '        ]'
2026-03-14 09:44:27.436545 | controller |     - '    logger.go:42: 09:42:08 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.436549 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.436553 | controller |     - '    logger.go:42: 09:42:08 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.436557 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.436561 | controller |     - '    logger.go:42: 09:42:08 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.436566 | controller |     - '    logger.go:42: 09:42:09 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.436570 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.436574 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.436578 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.436582 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.436589 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.436593 | controller |     - '        ]'
2026-03-14 09:44:27.436598 | controller |     - '    logger.go:42: 09:42:09 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.436602 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.436607 | controller |     - '    logger.go:42: 09:42:09 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.436612 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.436616 | controller |     - '    logger.go:42: 09:42:09 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.436620 | controller |     - '    logger.go:42: 09:42:10 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.436624 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.436628 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.436634 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.436638 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.436642 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.436646 | controller |     - '        ]'
2026-03-14 09:44:27.436651 | controller |     - '    logger.go:42: 09:42:10 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.436655 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.436665 | controller |     - '    logger.go:42: 09:42:10 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.436670 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.436674 | controller |     - '    logger.go:42: 09:42:11 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.436678 | controller |     - '    logger.go:42: 09:42:12 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.436682 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.436686 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.436691 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.436695 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.436699 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.436703 | controller |     - '        ]'
2026-03-14 09:44:27.436707 | controller |     - '    logger.go:42: 09:42:12 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.436711 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.436716 | controller |     - '    logger.go:42: 09:42:12 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.436720 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.436724 | controller |     - '    logger.go:42: 09:42:12 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.436728 | controller |     - '    logger.go:42: 09:42:13 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.436732 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.436736 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.436741 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.436745 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.436749 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.436753 | controller |     - '        ]'
2026-03-14 09:44:27.436757 | controller |     - '    logger.go:42: 09:42:13 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.436761 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.436766 | controller |     - '    logger.go:42: 09:42:13 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.436770 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.436774 | controller |     - '    logger.go:42: 09:42:13 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.436778 | controller |     - '    logger.go:42: 09:42:14 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.436782 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.436787 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.436791 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.436795 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.436799 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.436803 | controller |     - '        ]'
2026-03-14 09:44:27.436807 | controller |     - '    logger.go:42: 09:42:14 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.436812 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.436817 | controller |     - '    logger.go:42: 09:42:14 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.436822 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.436826 | controller |     - '    logger.go:42: 09:42:14 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.436830 | controller |     - '    logger.go:42: 09:42:15 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.436834 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.436838 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.436843 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.436847 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.436851 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.436855 | controller |     - '        ]'
2026-03-14 09:44:27.436859 | controller |     - '    logger.go:42: 09:42:15 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.436863 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.436868 | controller |     - '    logger.go:42: 09:42:15 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.436872 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.436876 | controller |     - '    logger.go:42: 09:42:15 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.436880 | controller |     - '    logger.go:42: 09:42:16 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.436884 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.436889 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.436895 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.436900 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.436906 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.436912 | controller |     - '        ]'
2026-03-14 09:44:27.436929 | controller |     - '    logger.go:42: 09:42:16 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.436937 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.436943 | controller |     - '    logger.go:42: 09:42:16 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.436951 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.436957 | controller |     - '    logger.go:42: 09:42:16 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.436962 | controller |     - '    logger.go:42: 09:42:17 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.436968 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.436974 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.436980 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.436986 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.436991 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.436997 | controller |     - '        ]'
2026-03-14 09:44:27.437003 | controller |     - '    logger.go:42: 09:42:17 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.437009 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.437015 | controller |     - '    logger.go:42: 09:42:17 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.437020 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.437026 | controller |     - '    logger.go:42: 09:42:18 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.437032 | controller |     - '    logger.go:42: 09:42:19 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.437041 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.437047 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.437052 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.437058 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.437064 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.437070 | controller |     - '        ]'
2026-03-14 09:44:27.437075 | controller |     - '    logger.go:42: 09:42:19 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.437081 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.437087 | controller |     - '    logger.go:42: 09:42:19 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.437093 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.437099 | controller |     - '    logger.go:42: 09:42:19 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.437105 | controller |     - '    logger.go:42: 09:42:20 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.437110 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.437116 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.437122 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.437127 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.437133 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.437139 | controller |     - '        ]'
2026-03-14 09:44:27.437144 | controller |     - '    logger.go:42: 09:42:20 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.437150 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.437156 | controller |     - '    logger.go:42: 09:42:20 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.437162 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.437167 | controller |     - '    logger.go:42: 09:42:20 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.437173 | controller |     - '    logger.go:42: 09:42:21 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.437178 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.437184 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.437190 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.437195 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.437200 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.437206 | controller |     - '        ]'
2026-03-14 09:44:27.437212 | controller |     - '    logger.go:42: 09:42:21 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.437218 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.437224 | controller |     - '    logger.go:42: 09:42:21 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.437229 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.437235 | controller |     - '    logger.go:42: 09:42:21 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.437241 | controller |     - '    logger.go:42: 09:42:22 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.437247 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.437253 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.437259 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.437264 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.437279 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.437286 | controller |     - '        ]'
2026-03-14 09:44:27.437316 | controller |     - '    logger.go:42: 09:42:22 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.437323 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.437329 | controller |     - '    logger.go:42: 09:42:22 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.437335 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.437341 | controller |     - '    logger.go:42: 09:42:22 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.437347 | controller |     - '    logger.go:42: 09:42:23 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.437353 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.437358 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.437364 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.437369 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.437378 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.437384 | controller |     - '        ]'
2026-03-14 09:44:27.437390 | controller |     - '    logger.go:42: 09:42:23 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.437395 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.437401 | controller |     - '    logger.go:42: 09:42:23 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.437406 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.437412 | controller |     - '    logger.go:42: 09:42:24 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.437421 | controller |     - '    logger.go:42: 09:42:25 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.437427 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.437433 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.437438 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.437444 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.437450 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.437456 | controller |     - '        ]'
2026-03-14 09:44:27.437462 | controller |     - '    logger.go:42: 09:42:25 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.437468 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.437474 | controller |     - '    logger.go:42: 09:42:25 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.437481 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.437486 | controller |     - '    logger.go:42: 09:42:25 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.437492 | controller |     - '    logger.go:42: 09:42:26 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.437498 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.437505 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.437511 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.437517 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.437522 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.437528 | controller |     - '        ]'
2026-03-14 09:44:27.437534 | controller |     - '    logger.go:42: 09:42:26 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.437540 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.437546 | controller |     - '    logger.go:42: 09:42:26 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.437551 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.437557 | controller |     - '    logger.go:42: 09:42:26 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.437565 | controller |     - '    logger.go:42: 09:42:27 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.437569 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.437577 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.437583 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.437587 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.437591 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.437596 | controller |     - '        ]'
2026-03-14 09:44:27.437601 | controller |     - '    logger.go:42: 09:42:27 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.437606 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.437610 | controller |     - '    logger.go:42: 09:42:27 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.437615 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.437619 | controller |     - '    logger.go:42: 09:42:27 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.437624 | controller |     - '    logger.go:42: 09:42:28 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.437630 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.437648 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.437655 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.437674 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.437681 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.437686 | controller |     - '        ]'
2026-03-14 09:44:27.437692 | controller |     - '    logger.go:42: 09:42:28 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.437698 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.437703 | controller |     - '    logger.go:42: 09:42:28 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.437709 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.437727 | controller |     - '    logger.go:42: 09:42:28 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.437733 | controller |     - '    logger.go:42: 09:42:29 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.437738 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.437744 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.437750 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.437756 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.437761 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.437768 | controller |     - '        ]'
2026-03-14 09:44:27.437773 | controller |     - '    logger.go:42: 09:42:29 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.437779 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.437784 | controller |     - '    logger.go:42: 09:42:29 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.437790 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.437795 | controller |     - '    logger.go:42: 09:42:30 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.437801 | controller |     - '    logger.go:42: 09:42:31 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.437806 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.437812 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.437816 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.437820 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.437827 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.437832 | controller |     - '        ]'
2026-03-14 09:44:27.437836 | controller |     - '    logger.go:42: 09:42:31 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.437840 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.437844 | controller |     - '    logger.go:42: 09:42:31 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.437848 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.437853 | controller |     - '    logger.go:42: 09:42:31 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.437857 | controller |     - '    logger.go:42: 09:42:32 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.437861 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.437867 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.437873 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.437882 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.437896 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.437902 | controller |     - '        ]'
2026-03-14 09:44:27.437907 | controller |     - '    logger.go:42: 09:42:32 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.437913 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.437918 | controller |     - '    logger.go:42: 09:42:32 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.437925 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.437932 | controller |     - '    logger.go:42: 09:42:32 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.437938 | controller |     - '    logger.go:42: 09:42:33 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.437946 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.437952 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.437958 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.437962 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.437966 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.437971 | controller |     - '        ]'
2026-03-14 09:44:27.437975 | controller |     - '    logger.go:42: 09:42:33 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.437979 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.437983 | controller |     - '    logger.go:42: 09:42:33 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.437987 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.437992 | controller |     - '    logger.go:42: 09:42:33 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.437996 | controller |     - '    logger.go:42: 09:42:34 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.438000 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.438008 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.438012 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.438017 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.438021 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.438025 | controller |     - '        ]'
2026-03-14 09:44:27.438029 | controller |     - '    logger.go:42: 09:42:34 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.438033 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.438037 | controller |     - '    logger.go:42: 09:42:34 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.438045 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.438049 | controller |     - '    logger.go:42: 09:42:34 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.438053 | controller |     - '    logger.go:42: 09:42:35 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.438057 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.438061 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.438066 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.438070 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.438074 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.438078 | controller |     - '        ]'
2026-03-14 09:44:27.438082 | controller |     - '    logger.go:42: 09:42:35 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.438086 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.438090 | controller |     - '    logger.go:42: 09:42:35 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.438094 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.438099 | controller |     - '    logger.go:42: 09:42:36 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.438103 | controller |     - '    logger.go:42: 09:42:37 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.438107 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.438111 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.438115 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.438119 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.438123 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.438128 | controller |     - '        ]'
2026-03-14 09:44:27.438132 | controller |     - '    logger.go:42: 09:42:37 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.438136 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.438140 | controller |     - '    logger.go:42: 09:42:37 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.438144 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.438148 | controller |     - '    logger.go:42: 09:42:37 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.438152 | controller |     - '    logger.go:42: 09:42:38 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.438157 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.438161 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.438165 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.438169 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.438187 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.438191 | controller |     - '        ]'
2026-03-14 09:44:27.438196 | controller |     - '    logger.go:42: 09:42:38 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.438200 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.438204 | controller |     - '    logger.go:42: 09:42:38 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.438235 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.438241 | controller |     - '    logger.go:42: 09:42:38 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.438246 | controller |     - '    logger.go:42: 09:42:39 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.438252 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.438257 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.438265 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.438273 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.438278 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.438284 | controller |     - '        ]'
2026-03-14 09:44:27.438289 | controller |     - '    logger.go:42: 09:42:39 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.438322 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.438328 | controller |     - '    logger.go:42: 09:42:39 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.438333 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.438338 | controller |     - '    logger.go:42: 09:42:39 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.438348 | controller |     - '    logger.go:42: 09:42:40 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.438354 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.438359 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.438365 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.438371 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.438376 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.438382 | controller |     - '        ]'
2026-03-14 09:44:27.438387 | controller |     - '    logger.go:42: 09:42:40 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.438393 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.438398 | controller |     - '    logger.go:42: 09:42:40 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.438404 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.438410 | controller |     - '    logger.go:42: 09:42:40 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.438415 | controller |     - '    logger.go:42: 09:42:41 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.438420 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.438425 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.438431 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.438436 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.438442 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.438447 | controller |     - '        ]'
2026-03-14 09:44:27.438453 | controller |     - '    logger.go:42: 09:42:41 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.438458 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.438464 | controller |     - '    logger.go:42: 09:42:41 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.438470 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.438475 | controller |     - '    logger.go:42: 09:42:42 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.438481 | controller |     - '    logger.go:42: 09:42:43 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.438485 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.438489 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.438493 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.438497 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.438503 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.438508 | controller |     - '        ]'
2026-03-14 09:44:27.438513 | controller |     - '    logger.go:42: 09:42:43 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.438519 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.438528 | controller |     - '    logger.go:42: 09:42:43 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.438533 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.438539 | controller |     - '    logger.go:42: 09:42:43 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.438544 | controller |     - '    logger.go:42: 09:42:44 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.438549 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.438553 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.438558 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.438562 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.438568 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.438573 | controller |     - '        ]'
2026-03-14 09:44:27.438578 | controller |     - '    logger.go:42: 09:42:44 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.438584 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.438588 | controller |     - '    logger.go:42: 09:42:44 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.438592 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.438597 | controller |     - '    logger.go:42: 09:42:44 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.438603 | controller |     - '    logger.go:42: 09:42:45 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.438609 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.438615 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.438621 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.438627 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.438632 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.438638 | controller |     - '        ]'
2026-03-14 09:44:27.438643 | controller |     - '    logger.go:42: 09:42:45 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.438649 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.438654 | controller |     - '    logger.go:42: 09:42:45 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.438664 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.438670 | controller |     - '    logger.go:42: 09:42:45 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.438675 | controller |     - '    logger.go:42: 09:42:46 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.438681 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.438686 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.438692 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.438698 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.438703 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.438709 | controller |     - '        ]'
2026-03-14 09:44:27.438714 | controller |     - '    logger.go:42: 09:42:46 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.438720 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.438729 | controller |     - '    logger.go:42: 09:42:46 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.438735 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.438739 | controller |     - '    logger.go:42: 09:42:46 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']'''
2026-03-14 09:44:27.438743 | controller |     - '    logger.go:42: 09:42:47 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.438748 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.438756 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.438761 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.438767 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.438771 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.438775 | controller |     - '        ]'
2026-03-14 09:44:27.438780 | controller |     - '    logger.go:42: 09:42:47 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.438784 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.438788 | controller |     - '    logger.go:42: 09:42:47 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.438792 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.438797 | controller |     - '    logger.go:42: 09:42:48 | watcher-cinder/2-deploy-cinder | Error from server
2026-03-14 09:44:27.438801 | controller |       (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0"
2026-03-14 09:44:27.438805 | controller |       is waiting to start: ContainerCreating'
2026-03-14 09:44:27.438810 | controller |     - '    logger.go:42: 09:42:48 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:44:27.438815 | controller |     - '    logger.go:42: 09:42:49 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.438819 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.438823 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.438827 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.438832 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.438836 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.438840 | controller |     - '        ]'
2026-03-14 09:44:27.438844 | controller |     - '    logger.go:42: 09:42:49 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.438848 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.438853 | controller |     - '    logger.go:42: 09:42:49 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.438857 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.438861 | controller |     - '    logger.go:42: 09:42:49 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:44:27.438865 | controller |     - '    logger.go:42: 09:42:50 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.438869 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.438874 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.438878 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.438882 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.438886 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.438890 | controller |     - '        ]'
2026-03-14 09:44:27.438895 | controller |     - '    logger.go:42: 09:42:50 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.438899 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.438903 | controller |     - '    logger.go:42: 09:42:50 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.438907 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.438912 | controller |     - '    logger.go:42: 09:42:50 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:44:27.438916 | controller |     - '    logger.go:42: 09:42:51 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.438920 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.438924 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.438928 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.438934 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.438942 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.438946 | controller |     - '        ]'
2026-03-14 09:44:27.438951 | controller |     - '    logger.go:42: 09:42:51 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.438955 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.438959 | controller |     - '    logger.go:42: 09:42:51 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.438963 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.438968 | controller |     - '    logger.go:42: 09:42:51 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:44:27.438972 | controller |     - '    logger.go:42: 09:42:52 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.438976 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.438980 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.438984 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.438989 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.438995 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.438999 | controller |     - '        ]'
2026-03-14 09:44:27.439003 | controller |     - '    logger.go:42: 09:42:52 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.439008 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.439012 | controller |     - '    logger.go:42: 09:42:52 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.439016 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.439020 | controller |     - '    logger.go:42: 09:42:52 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:44:27.439025 | controller |     - '    logger.go:42: 09:42:54 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.439029 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.439033 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.439037 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.439042 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.439046 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.439050 | controller |     - '        ]'
2026-03-14 09:44:27.439054 | controller |     - '    logger.go:42: 09:42:54 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.439059 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.439067 | controller |     - '    logger.go:42: 09:42:54 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.439071 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.439076 | controller |     - '    logger.go:42: 09:42:54 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:44:27.439081 | controller |     - '    logger.go:42: 09:42:55 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.439085 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.439089 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.439093 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.439098 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.439102 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.439106 | controller |     - '        ]'
2026-03-14 09:44:27.439110 | controller |     - '    logger.go:42: 09:42:55 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.439114 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.439120 | controller |     - '    logger.go:42: 09:42:55 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.439125 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.439129 | controller |     - '    logger.go:42: 09:42:55 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:44:27.439133 | controller |     - '    logger.go:42: 09:42:56 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.439138 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.439142 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.439146 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.439150 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.439155 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.439159 | controller |     - '        ]'
2026-03-14 09:44:27.439163 | controller |     - '    logger.go:42: 09:42:56 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.439167 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.439174 | controller |     - '    logger.go:42: 09:42:56 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.439178 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.439182 | controller |     - '    logger.go:42: 09:42:56 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:44:27.439186 | controller |     - '    logger.go:42: 09:42:57 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.439191 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.439195 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.439202 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.439207 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.439211 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.439215 | controller |     - '        ]'
2026-03-14 09:44:27.439220 | controller |     - '    logger.go:42: 09:42:57 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.439224 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.439228 | controller |     - '    logger.go:42: 09:42:57 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.439232 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.439237 | controller |     - '    logger.go:42: 09:42:58 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:44:27.439241 | controller |     - '    logger.go:42: 09:42:59 | watcher-cinder/2-deploy-cinder | running command:
2026-03-14 09:44:27.439245 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.439249 | controller |     - '        # check that the decision detects that there is a cinder service and'
2026-03-14 09:44:27.439253 | controller |     - '        # does not log that storage collector is skipped'
2026-03-14 09:44:27.439258 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.439262 | controller |       storage service is not enabled, skipping storage collector'')" == 0 ]'
2026-03-14 09:44:27.439266 | controller |     - '        ]'
2026-03-14 09:44:27.439271 | controller |     - '    logger.go:42: 09:42:59 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.439275 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.439279 | controller |     - '    logger.go:42: 09:42:59 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.439283 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.439319 | controller |     - '    logger.go:42: 09:42:59 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']'''
2026-03-14 09:44:27.439332 | controller |     - '    logger.go:42: 09:42:59 | watcher-cinder/2-deploy-cinder | test step completed
2026-03-14 09:44:27.439337 | controller |       2-deploy-cinder'
2026-03-14 09:44:27.439343 | controller |     - '    logger.go:42: 09:42:59 | watcher-cinder/3-remove-cinder | starting test step
2026-03-14 09:44:27.439348 | controller |       3-remove-cinder'
2026-03-14 09:44:27.439352 | controller |     - '    logger.go:42: 09:42:59 | watcher-cinder/3-remove-cinder | OpenStackControlPlane:watcher-kuttl-default/openstack
2026-03-14 09:44:27.439356 | controller |       updated'
2026-03-14 09:44:27.439360 | controller |     - '    logger.go:42: 09:42:59 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:44:27.439365 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.439369 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.439373 | controller |       service'
2026-03-14 09:44:27.439378 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.439382 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.439386 | controller |     - '        ]'
2026-03-14 09:44:27.439391 | controller |     - '    logger.go:42: 09:42:59 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.439395 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.439399 | controller |     - '    logger.go:42: 09:42:59 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.439403 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.439407 | controller |     - '    logger.go:42: 09:42:59 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:44:27.439412 | controller |     - '    logger.go:42: 09:43:00 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:44:27.439416 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.439420 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.439425 | controller |       service'
2026-03-14 09:44:27.439432 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.439436 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.439441 | controller |     - '        ]'
2026-03-14 09:44:27.439445 | controller |     - '    logger.go:42: 09:43:00 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.439450 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.439454 | controller |     - '    logger.go:42: 09:43:00 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.439458 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.439464 | controller |     - '    logger.go:42: 09:43:00 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:44:27.439468 | controller |     - '    logger.go:42: 09:43:01 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:44:27.439472 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.439477 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.439481 | controller |       service'
2026-03-14 09:44:27.439486 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.439490 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.439494 | controller |     - '        ]'
2026-03-14 09:44:27.439498 | controller |     - '    logger.go:42: 09:43:01 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.439503 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.439507 | controller |     - '    logger.go:42: 09:43:01 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.439515 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.439520 | controller |     - '    logger.go:42: 09:43:01 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:44:27.439525 | controller |     - '    logger.go:42: 09:43:02 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:44:27.439529 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.439533 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.439539 | controller |       service'
2026-03-14 09:44:27.439543 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.439547 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.439552 | controller |     - '        ]'
2026-03-14 09:44:27.439556 | controller |     - '    logger.go:42: 09:43:02 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.439560 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.439564 | controller |     - '    logger.go:42: 09:43:02 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.439568 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.439573 | controller |     - '    logger.go:42: 09:43:03 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:44:27.439577 | controller |     - '    logger.go:42: 09:43:04 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:44:27.439581 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.439585 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.439589 | controller |       service'
2026-03-14 09:44:27.439594 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.439598 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.439602 | controller |     - '        ]'
2026-03-14 09:44:27.439606 | controller |     - '    logger.go:42: 09:43:04 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.439611 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.439615 | controller |     - '    logger.go:42: 09:43:04 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.439619 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.439623 | controller |     - '    logger.go:42: 09:43:04 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:44:27.439627 | controller |     - '    logger.go:42: 09:43:05 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:44:27.439632 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.439636 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.439640 | controller |       service'
2026-03-14 09:44:27.439644 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.439648 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.439653 | controller |     - '        ]'
2026-03-14 09:44:27.439657 | controller |     - '    logger.go:42: 09:43:05 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.439661 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.439665 | controller |     - '    logger.go:42: 09:43:05 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.439670 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.439674 | controller |     - '    logger.go:42: 09:43:05 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:44:27.439678 | controller |     - '    logger.go:42: 09:43:06 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:44:27.439682 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.439686 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.439691 | controller |       service'
2026-03-14 09:44:27.439695 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.439699 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.439703 | controller |     - '        ]'
2026-03-14 09:44:27.439707 | controller |     - '    logger.go:42: 09:43:06 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.439712 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.439716 | controller |     - '    logger.go:42: 09:43:06 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.439721 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.439726 | controller |     - '    logger.go:42: 09:43:06 | watcher-cinder/3-remove-cinder | Error from server
2026-03-14 09:44:27.439730 | controller |       (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0"
2026-03-14 09:44:27.439734 | controller |       is waiting to start: ContainerCreating'
2026-03-14 09:44:27.439738 | controller |     - '    logger.go:42: 09:43:06 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:44:27.439742 | controller |     - '    logger.go:42: 09:43:07 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:44:27.439747 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.439751 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.439755 | controller |       service'
2026-03-14 09:44:27.439761 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.439765 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.439769 | controller |     - '        ]'
2026-03-14 09:44:27.439774 | controller |     - '    logger.go:42: 09:43:07 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.439780 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.439785 | controller |     - '    logger.go:42: 09:43:07 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.439789 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.439793 | controller |     - '    logger.go:42: 09:43:07 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:44:27.439797 | controller |     - '    logger.go:42: 09:43:08 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:44:27.439802 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.439806 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.439810 | controller |       service'
2026-03-14 09:44:27.439814 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.439818 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.439822 | controller |     - '        ]'
2026-03-14 09:44:27.439827 | controller |     - '    logger.go:42: 09:43:08 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.439831 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.439835 | controller |     - '    logger.go:42: 09:43:08 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.439839 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.439844 | controller |     - '    logger.go:42: 09:43:08 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:44:27.439848 | controller |     - '    logger.go:42: 09:43:09 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:44:27.439852 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.439856 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.439860 | controller |       service'
2026-03-14 09:44:27.439864 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.439868 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.439873 | controller |     - '        ]'
2026-03-14 09:44:27.439877 | controller |     - '    logger.go:42: 09:43:09 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.439881 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.439887 | controller |     - '    logger.go:42: 09:43:09 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.439891 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.439895 | controller |     - '    logger.go:42: 09:43:10 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']'''
2026-03-14 09:44:27.439899 | controller |     - '    logger.go:42: 09:43:11 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:44:27.439905 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.439909 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.439914 | controller |       service'
2026-03-14 09:44:27.439918 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.439922 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.439926 | controller |     - '        ]'
2026-03-14 09:44:27.439930 | controller |     - '    logger.go:42: 09:43:11 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.439935 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.439939 | controller |     - '    logger.go:42: 09:43:11 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.439943 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.439947 | controller |     - '    logger.go:42: 09:43:11 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']'''
2026-03-14 09:44:27.439952 | controller |     - '    logger.go:42: 09:43:12 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:44:27.439956 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.439960 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.439965 | controller |       service'
2026-03-14 09:44:27.439969 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.439973 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.439977 | controller |     - '        ]'
2026-03-14 09:44:27.439982 | controller |     - '    logger.go:42: 09:43:12 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.439986 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.439990 | controller |     - '    logger.go:42: 09:43:12 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.439994 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.439998 | controller |     - '    logger.go:42: 09:43:12 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']'''
2026-03-14 09:44:27.440003 | controller |     - '    logger.go:42: 09:43:13 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:44:27.440007 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.440011 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.440015 | controller |       service'
2026-03-14 09:44:27.440019 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.440024 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.440028 | controller |     - '        ]'
2026-03-14 09:44:27.440032 | controller |     - '    logger.go:42: 09:43:13 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.440036 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.440042 | controller |     - '    logger.go:42: 09:43:13 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.440047 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.440051 | controller |     - '    logger.go:42: 09:43:13 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']'''
2026-03-14 09:44:27.440055 | controller |     - '    logger.go:42: 09:43:14 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:44:27.440059 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.440063 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.440068 | controller |       service'
2026-03-14 09:44:27.440072 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.440076 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.440080 | controller |     - '        ]'
2026-03-14 09:44:27.440086 | controller |     - '    logger.go:42: 09:43:14 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.440090 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.440096 | controller |     - '    logger.go:42: 09:43:14 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.440100 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.440105 | controller |     - '    logger.go:42: 09:43:14 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']'''
2026-03-14 09:44:27.440109 | controller |     - '    logger.go:42: 09:43:15 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:44:27.440113 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.440118 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.440122 | controller |       service'
2026-03-14 09:44:27.440126 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.440130 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.440134 | controller |     - '        ]'
2026-03-14 09:44:27.440139 | controller |     - '    logger.go:42: 09:43:15 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.440143 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.440147 | controller |     - '    logger.go:42: 09:43:15 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.440151 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.440157 | controller |     - '    logger.go:42: 09:43:16 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']'''
2026-03-14 09:44:27.440162 | controller |     - '    logger.go:42: 09:43:17 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:44:27.440166 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.440170 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.440174 | controller |       service'
2026-03-14 09:44:27.440179 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.440183 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.440187 | controller |     - '        ]'
2026-03-14 09:44:27.440191 | controller |     - '    logger.go:42: 09:43:17 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.440195 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.440200 | controller |     - '    logger.go:42: 09:43:17 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.440204 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.440209 | controller |     - '    logger.go:42: 09:43:17 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']'''
2026-03-14 09:44:27.440214 | controller |     - '    logger.go:42: 09:43:18 | watcher-cinder/3-remove-cinder | running command:
2026-03-14 09:44:27.440219 | controller |       [sh -c set -euxo pipefail'
2026-03-14 09:44:27.440225 | controller |     - '        # check that the decision engine correctly detects that there is no cinder
2026-03-14 09:44:27.440230 | controller |       service'
2026-03-14 09:44:27.440235 | controller |     - '        [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block
2026-03-14 09:44:27.440241 | controller |       storage service is not enabled, skipping storage collector'')" == 2 ]'
2026-03-14 09:44:27.440246 | controller |     - '        ]'
2026-03-14 09:44:27.440252 | controller |     - '    logger.go:42: 09:43:18 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default
2026-03-14 09:44:27.440258 | controller |       watcher-kuttl-decision-engine-0'
2026-03-14 09:44:27.440263 | controller |     - '    logger.go:42: 09:43:18 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block
2026-03-14 09:44:27.440269 | controller |       storage service is not enabled, skipping storage collector'''
2026-03-14 09:44:27.440275 | controller |     - '    logger.go:42: 09:43:18 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']'''
2026-03-14 09:44:27.440280 | controller |     - '    logger.go:42: 09:43:18 | watcher-cinder/3-remove-cinder | test step completed
2026-03-14 09:44:27.440286 | controller |       3-remove-cinder'
2026-03-14 09:44:27.440313 | controller |     - '    logger.go:42: 09:43:18 | watcher-cinder/4-cleanup-watcher | starting test step
2026-03-14 09:44:27.440321 | controller |       4-cleanup-watcher'
2026-03-14 09:44:27.440327 | controller |     - '    logger.go:42: 09:43:30 | watcher-cinder/4-cleanup-watcher | test step completed
2026-03-14 09:44:27.440333 | controller |       4-cleanup-watcher'
2026-03-14 09:44:27.440338 | controller |     - '    logger.go:42: 09:43:30 | watcher-cinder | skipping kubernetes event logging'
2026-03-14 09:44:27.440343 | controller |     - === CONT  kuttl/harness/watcher-api-scaling
2026-03-14 09:44:27.440349 | controller |     - '    logger.go:42: 09:43:30 | watcher-api-scaling | Skipping creation of user-supplied
2026-03-14 09:44:27.440354 | controller |       namespace: watcher-kuttl-default'
2026-03-14 09:44:27.440360 | controller |     - '    logger.go:42: 09:43:30 | watcher-api-scaling/0-cleanup-watcher | starting test
2026-03-14 09:44:27.440370 | controller |       step 0-cleanup-watcher'
2026-03-14 09:44:27.440376 | controller |     - '    logger.go:42: 09:43:30 | watcher-api-scaling/0-cleanup-watcher | test step
2026-03-14 09:44:27.440382 | controller |       completed 0-cleanup-watcher'
2026-03-14 09:44:27.440388 | controller |     - '    logger.go:42: 09:43:30 | watcher-api-scaling/1-deploy-with-defaults | starting
2026-03-14 09:44:27.440394 | controller |       test step 1-deploy-with-defaults'
2026-03-14 09:44:27.440399 | controller |     - '    logger.go:42: 09:43:30 | watcher-api-scaling/1-deploy-with-defaults | Watcher:watcher-kuttl-default/watcher-kuttl
2026-03-14 09:44:27.440405 | controller |       created'
2026-03-14 09:44:27.440410 | controller |     - '    logger.go:42: 09:43:30 | watcher-api-scaling/1-deploy-with-defaults | running
2026-03-14 09:44:27.440416 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.440422 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.440427 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.440433 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:44:27.440438 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.440444 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.440449 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.440455 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.440461 | controller |       ]'
2026-03-14 09:44:27.440466 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.440472 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.440477 | controller |       == "" ]; then'
2026-03-14 09:44:27.440483 | controller |     - '            exit 0'
2026-03-14 09:44:27.440488 | controller |     - '        fi'
2026-03-14 09:44:27.440494 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.440499 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.440504 | controller |     - '        counter=0'
2026-03-14 09:44:27.440509 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.440515 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.440520 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.440525 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.440530 | controller |     - '            fi'
2026-03-14 09:44:27.440535 | controller |     - '        done'
2026-03-14 09:44:27.440541 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.440547 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.440552 | controller |     - '          exit 1'
2026-03-14 09:44:27.440558 | controller |     - '        else'
2026-03-14 09:44:27.440563 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.440569 | controller |     - '        fi'
2026-03-14 09:44:27.440575 | controller |     - '        ]'
2026-03-14 09:44:27.440580 | controller |     - '    logger.go:42: 09:43:30 | watcher-api-scaling/1-deploy-with-defaults | + oc
2026-03-14 09:44:27.440594 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.440600 | controller |       -c Name -c Type'
2026-03-14 09:44:27.440606 | controller |     - '    logger.go:42: 09:43:30 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:44:27.440612 | controller |       -c ''^watcher'''
2026-03-14 09:44:27.440618 | controller |     - '    logger.go:42: 09:43:32 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.440624 | controller |       0 == 1 '']'''
2026-03-14 09:44:27.440632 | controller |     - '    logger.go:42: 09:43:33 | watcher-api-scaling/1-deploy-with-defaults | running
2026-03-14 09:44:27.440638 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.440644 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.440650 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.440655 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:44:27.440661 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.440666 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.440672 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.440677 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.440683 | controller |       ]'
2026-03-14 09:44:27.440689 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.440694 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.440700 | controller |       == "" ]; then'
2026-03-14 09:44:27.440705 | controller |     - '            exit 0'
2026-03-14 09:44:27.440710 | controller |     - '        fi'
2026-03-14 09:44:27.440716 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.440721 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.440726 | controller |     - '        counter=0'
2026-03-14 09:44:27.440732 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.440738 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.440743 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.440749 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.440754 | controller |     - '            fi'
2026-03-14 09:44:27.440760 | controller |     - '        done'
2026-03-14 09:44:27.440765 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.440774 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.440780 | controller |     - '          exit 1'
2026-03-14 09:44:27.440785 | controller |     - '        else'
2026-03-14 09:44:27.440796 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.440801 | controller |     - '        fi'
2026-03-14 09:44:27.440807 | controller |     - '        ]'
2026-03-14 09:44:27.440812 | controller |     - '    logger.go:42: 09:43:33 | watcher-api-scaling/1-deploy-with-defaults | + oc
2026-03-14 09:44:27.440818 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.440823 | controller |       -c Name -c Type'
2026-03-14 09:44:27.440829 | controller |     - '    logger.go:42: 09:43:33 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:44:27.440834 | controller |       -c ''^watcher'''
2026-03-14 09:44:27.440840 | controller |     - '    logger.go:42: 09:43:35 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.440845 | controller |       0 == 1 '']'''
2026-03-14 09:44:27.440851 | controller |     - '    logger.go:42: 09:43:37 | watcher-api-scaling/1-deploy-with-defaults | running
2026-03-14 09:44:27.440856 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.440862 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.440867 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.440876 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:44:27.440881 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.440887 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.440892 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.440898 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.440903 | controller |       ]'
2026-03-14 09:44:27.440908 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.440914 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.440919 | controller |       == "" ]; then'
2026-03-14 09:44:27.440925 | controller |     - '            exit 0'
2026-03-14 09:44:27.440931 | controller |     - '        fi'
2026-03-14 09:44:27.440937 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.440943 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.440948 | controller |     - '        counter=0'
2026-03-14 09:44:27.440953 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.440959 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.440964 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.440969 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.440975 | controller |     - '            fi'
2026-03-14 09:44:27.440981 | controller |     - '        done'
2026-03-14 09:44:27.440986 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.440991 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.440997 | controller |     - '          exit 1'
2026-03-14 09:44:27.441002 | controller |     - '        else'
2026-03-14 09:44:27.441008 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.441013 | controller |     - '        fi'
2026-03-14 09:44:27.441018 | controller |     - '        ]'
2026-03-14 09:44:27.441024 | controller |     - '    logger.go:42: 09:43:37 | watcher-api-scaling/1-deploy-with-defaults | + oc
2026-03-14 09:44:27.441029 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.441035 | controller |       -c Name -c Type'
2026-03-14 09:44:27.441040 | controller |     - '    logger.go:42: 09:43:37 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:44:27.441046 | controller |       -c ''^watcher'''
2026-03-14 09:44:27.441051 | controller |     - '    logger.go:42: 09:43:39 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.441056 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.441062 | controller |     - '    logger.go:42: 09:43:39 | watcher-api-scaling/1-deploy-with-defaults | ++ awk
2026-03-14 09:44:27.441067 | controller |       ''{print $1}'''
2026-03-14 09:44:27.441073 | controller |     - '    logger.go:42: 09:43:39 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:44:27.441078 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.441084 | controller |       -c Name -c Type -c ID'
2026-03-14 09:44:27.441089 | controller |     - '    logger.go:42: 09:43:39 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:44:27.441095 | controller |       watcher'
2026-03-14 09:44:27.441100 | controller |     - '    logger.go:42: 09:43:41 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=86b564fd237a41c2b773bfe0f05b75f1'
2026-03-14 09:44:27.441106 | controller |     - '    logger.go:42: 09:43:41 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:44:27.441112 | controller |       get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.441117 | controller |     - '    logger.go:42: 09:43:41 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.441122 | controller |       86b564fd237a41c2b773bfe0f05b75f1 == 86b564fd237a41c2b773bfe0f05b75f1 '']'''
2026-03-14 09:44:27.441128 | controller |     - '    logger.go:42: 09:43:41 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:44:27.441136 | controller |       get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.441141 | controller |     - '    logger.go:42: 09:43:41 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.441147 | controller |       -n '''' '']'''
2026-03-14 09:44:27.441153 | controller |     - '    logger.go:42: 09:43:42 | watcher-api-scaling/1-deploy-with-defaults | running
2026-03-14 09:44:27.441158 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.441166 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.441177 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.441183 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:44:27.441188 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.441194 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.441199 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.441205 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.441211 | controller |       ]'
2026-03-14 09:44:27.441216 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.441222 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.441227 | controller |       == "" ]; then'
2026-03-14 09:44:27.441232 | controller |     - '            exit 0'
2026-03-14 09:44:27.441238 | controller |     - '        fi'
2026-03-14 09:44:27.441244 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.441250 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.441255 | controller |     - '        counter=0'
2026-03-14 09:44:27.441261 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.441266 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.441272 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.441277 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.441283 | controller |     - '            fi'
2026-03-14 09:44:27.441324 | controller |     - '        done'
2026-03-14 09:44:27.441339 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.441344 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.441350 | controller |     - '          exit 1'
2026-03-14 09:44:27.441356 | controller |     - '        else'
2026-03-14 09:44:27.441361 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.441367 | controller |     - '        fi'
2026-03-14 09:44:27.441373 | controller |     - '        ]'
2026-03-14 09:44:27.441378 | controller |     - '    logger.go:42: 09:43:42 | watcher-api-scaling/1-deploy-with-defaults | + oc
2026-03-14 09:44:27.441384 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.441390 | controller |       -c Name -c Type'
2026-03-14 09:44:27.441395 | controller |     - '    logger.go:42: 09:43:42 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:44:27.441401 | controller |       -c ''^watcher'''
2026-03-14 09:44:27.441407 | controller |     - '    logger.go:42: 09:43:44 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.441412 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.441418 | controller |     - '    logger.go:42: 09:43:44 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:44:27.441424 | controller |       watcher'
2026-03-14 09:44:27.441429 | controller |     - '    logger.go:42: 09:43:44 | watcher-api-scaling/1-deploy-with-defaults | ++ awk
2026-03-14 09:44:27.441435 | controller |       ''{print $1}'''
2026-03-14 09:44:27.441440 | controller |     - '    logger.go:42: 09:43:44 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:44:27.441445 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.441454 | controller |       -c Name -c Type -c ID'
2026-03-14 09:44:27.441459 | controller |     - '    logger.go:42: 09:43:46 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=86b564fd237a41c2b773bfe0f05b75f1'
2026-03-14 09:44:27.441465 | controller |     - '    logger.go:42: 09:43:46 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:44:27.441473 | controller |       get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.441479 | controller |     - '    logger.go:42: 09:43:47 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.441484 | controller |       86b564fd237a41c2b773bfe0f05b75f1 == 86b564fd237a41c2b773bfe0f05b75f1 '']'''
2026-03-14 09:44:27.441490 | controller |     - '    logger.go:42: 09:43:47 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:44:27.441496 | controller |       get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.441501 | controller |     - '    logger.go:42: 09:43:47 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.441507 | controller |       -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.441512 | controller |       '']'''
2026-03-14 09:44:27.441517 | controller |     - '    logger.go:42: 09:43:47 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:44:27.441523 | controller |       get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:44:27.441529 | controller |     - '    logger.go:42: 09:43:47 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.441534 | controller |       '''' == '''' '']'''
2026-03-14 09:44:27.441540 | controller |     - '    logger.go:42: 09:43:47 | watcher-api-scaling/1-deploy-with-defaults | + exit
2026-03-14 09:44:27.441545 | controller |       0'
2026-03-14 09:44:27.441551 | controller |     - '    logger.go:42: 09:43:48 | watcher-api-scaling/1-deploy-with-defaults | running
2026-03-14 09:44:27.441556 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.441562 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.441568 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.441573 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:44:27.441579 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.441586 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.441596 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.441600 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.441604 | controller |       ]'
2026-03-14 09:44:27.441609 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.441613 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.441618 | controller |       == "" ]; then'
2026-03-14 09:44:27.441624 | controller |     - '            exit 0'
2026-03-14 09:44:27.441630 | controller |     - '        fi'
2026-03-14 09:44:27.441636 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.441644 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.441650 | controller |     - '        counter=0'
2026-03-14 09:44:27.441656 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.441661 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.441667 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.441673 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.441679 | controller |     - '            fi'
2026-03-14 09:44:27.441685 | controller |     - '        done'
2026-03-14 09:44:27.441691 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.441697 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.441704 | controller |     - '          exit 1'
2026-03-14 09:44:27.441710 | controller |     - '        else'
2026-03-14 09:44:27.441716 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.441721 | controller |     - '        fi'
2026-03-14 09:44:27.441727 | controller |     - '        ]'
2026-03-14 09:44:27.441732 | controller |     - '    logger.go:42: 09:43:48 | watcher-api-scaling/1-deploy-with-defaults | + oc
2026-03-14 09:44:27.441737 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.441741 | controller |       -c Name -c Type'
2026-03-14 09:44:27.441745 | controller |     - '    logger.go:42: 09:43:48 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:44:27.441749 | controller |       -c ''^watcher'''
2026-03-14 09:44:27.441753 | controller |     - '    logger.go:42: 09:43:50 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.441758 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.441762 | controller |     - '    logger.go:42: 09:43:50 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:44:27.441766 | controller |       watcher'
2026-03-14 09:44:27.441772 | controller |     - '    logger.go:42: 09:43:50 | watcher-api-scaling/1-deploy-with-defaults | ++ awk
2026-03-14 09:44:27.441776 | controller |       ''{print $1}'''
2026-03-14 09:44:27.441790 | controller |     - '    logger.go:42: 09:43:50 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:44:27.441794 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.441798 | controller |       -c Name -c Type -c ID'
2026-03-14 09:44:27.441803 | controller |     - '    logger.go:42: 09:43:52 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=86b564fd237a41c2b773bfe0f05b75f1'
2026-03-14 09:44:27.441807 | controller |     - '    logger.go:42: 09:43:52 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:44:27.441811 | controller |       get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.441815 | controller |     - '    logger.go:42: 09:43:52 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.441820 | controller |       86b564fd237a41c2b773bfe0f05b75f1 == 86b564fd237a41c2b773bfe0f05b75f1 '']'''
2026-03-14 09:44:27.441824 | controller |     - '    logger.go:42: 09:43:52 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:44:27.441828 | controller |       get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.441832 | controller |     - '    logger.go:42: 09:43:52 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.441837 | controller |       -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.441841 | controller |       '']'''
2026-03-14 09:44:27.441845 | controller |     - '    logger.go:42: 09:43:52 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:44:27.441850 | controller |       get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:44:27.441854 | controller |     - '    logger.go:42: 09:43:52 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.441858 | controller |       '''' == '''' '']'''
2026-03-14 09:44:27.441862 | controller |     - '    logger.go:42: 09:43:52 | watcher-api-scaling/1-deploy-with-defaults | + exit
2026-03-14 09:44:27.441867 | controller |       0'
2026-03-14 09:44:27.441871 | controller |     - '    logger.go:42: 09:43:53 | watcher-api-scaling/1-deploy-with-defaults | running
2026-03-14 09:44:27.441875 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.441880 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.441884 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.441888 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:44:27.441893 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.441897 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.441901 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.441907 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.441911 | controller |       ]'
2026-03-14 09:44:27.441916 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.441923 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.441928 | controller |       == "" ]; then'
2026-03-14 09:44:27.441932 | controller |     - '            exit 0'
2026-03-14 09:44:27.441936 | controller |     - '        fi'
2026-03-14 09:44:27.441940 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.441945 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.441949 | controller |     - '        counter=0'
2026-03-14 09:44:27.441953 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.441958 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.441962 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.441966 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.441970 | controller |     - '            fi'
2026-03-14 09:44:27.441975 | controller |     - '        done'
2026-03-14 09:44:27.441979 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.441983 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.441988 | controller |     - '          exit 1'
2026-03-14 09:44:27.441992 | controller |     - '        else'
2026-03-14 09:44:27.441996 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.442000 | controller |     - '        fi'
2026-03-14 09:44:27.442004 | controller |     - '        ]'
2026-03-14 09:44:27.442009 | controller |     - '    logger.go:42: 09:43:53 | watcher-api-scaling/1-deploy-with-defaults | + oc
2026-03-14 09:44:27.442013 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.442017 | controller |       -c Name -c Type'
2026-03-14 09:44:27.442022 | controller |     - '    logger.go:42: 09:43:53 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:44:27.442026 | controller |       -c ''^watcher'''
2026-03-14 09:44:27.442030 | controller |     - '    logger.go:42: 09:43:55 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.442034 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.442039 | controller |     - '    logger.go:42: 09:43:55 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:44:27.442043 | controller |       watcher'
2026-03-14 09:44:27.442047 | controller |     - '    logger.go:42: 09:43:55 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:44:27.442052 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.442056 | controller |       -c Name -c Type -c ID'
2026-03-14 09:44:27.442060 | controller |     - '    logger.go:42: 09:43:55 | watcher-api-scaling/1-deploy-with-defaults | ++ awk
2026-03-14 09:44:27.442064 | controller |       ''{print $1}'''
2026-03-14 09:44:27.442069 | controller |     - '    logger.go:42: 09:43:58 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=86b564fd237a41c2b773bfe0f05b75f1'
2026-03-14 09:44:27.442073 | controller |     - '    logger.go:42: 09:43:58 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:44:27.442077 | controller |       get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.442081 | controller |     - '    logger.go:42: 09:43:58 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.442085 | controller |       86b564fd237a41c2b773bfe0f05b75f1 == 86b564fd237a41c2b773bfe0f05b75f1 '']'''
2026-03-14 09:44:27.442090 | controller |     - '    logger.go:42: 09:43:58 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:44:27.442094 | controller |       get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.442098 | controller |     - '    logger.go:42: 09:43:58 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.442102 | controller |       -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.442110 | controller |       '']'''
2026-03-14 09:44:27.442117 | controller |     - '    logger.go:42: 09:43:58 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:44:27.442123 | controller |       get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:44:27.442127 | controller |     - '    logger.go:42: 09:43:58 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.442132 | controller |       '''' == '''' '']'''
2026-03-14 09:44:27.442136 | controller |     - '    logger.go:42: 09:43:58 | watcher-api-scaling/1-deploy-with-defaults | + exit
2026-03-14 09:44:27.442140 | controller |       0'
2026-03-14 09:44:27.442144 | controller |     - '    logger.go:42: 09:43:59 | watcher-api-scaling/1-deploy-with-defaults | running
2026-03-14 09:44:27.442148 | controller |       command: [sh -c set -euxo pipefail'
2026-03-14 09:44:27.442153 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.442157 | controller |       -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]'
2026-03-14 09:44:27.442161 | controller |     - '        SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack
2026-03-14 09:44:27.442165 | controller |       service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')'
2026-03-14 09:44:27.442169 | controller |     - '        [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID})
2026-03-14 09:44:27.442174 | controller |       == ${SERVICEID} ]'
2026-03-14 09:44:27.442178 | controller |     - '        [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})"
2026-03-14 09:44:27.442182 | controller |       ]'
2026-03-14 09:44:27.442186 | controller |     - '        # If we are running the container locally, skip following test'
2026-03-14 09:44:27.442190 | controller |     - '        if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)"
2026-03-14 09:44:27.442195 | controller |       == "" ]; then'
2026-03-14 09:44:27.442380 | controller |     - '            exit 0'
2026-03-14 09:44:27.442390 | controller |     - '        fi'
2026-03-14 09:44:27.442394 | controller |     - '        env_variables=$(oc set env $(oc get pods -n openstack-operators -o name
2026-03-14 09:44:27.442398 | controller |       -l openstack.org/operator-name=watcher) -n openstack-operators --list)'
2026-03-14 09:44:27.442402 | controller |     - '        counter=0'
2026-03-14 09:44:27.442410 | controller |     - '        for i in ${env_variables}; do'
2026-03-14 09:44:27.442414 | controller |     - '            if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then'
2026-03-14 09:44:27.442419 | controller |     - '                echo ${i}'
2026-03-14 09:44:27.442423 | controller |     - '                counter=$((counter + 1))'
2026-03-14 09:44:27.442427 | controller |     - '            fi'
2026-03-14 09:44:27.442431 | controller |     - '        done'
2026-03-14 09:44:27.442436 | controller |     - '        if [ ${counter} -lt 3 ]; then'
2026-03-14 09:44:27.442440 | controller |     - '          echo "Error: Less than 3 _URL_DEFAULT variables found."'
2026-03-14 09:44:27.442444 | controller |     - '          exit 1'
2026-03-14 09:44:27.442448 | controller |     - '        else'
2026-03-14 09:44:27.442452 | controller |     - '          echo "Success: ${counter} _URL_DEFAULT variables found."'
2026-03-14 09:44:27.442457 | controller |     - '        fi'
2026-03-14 09:44:27.442461 | controller |     - '        ]'
2026-03-14 09:44:27.442465 | controller |     - '    logger.go:42: 09:43:59 | watcher-api-scaling/1-deploy-with-defaults | + oc
2026-03-14 09:44:27.442469 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.442473 | controller |       -c Name -c Type'
2026-03-14 09:44:27.442478 | controller |     - '    logger.go:42: 09:43:59 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:44:27.442482 | controller |       -c ''^watcher'''
2026-03-14 09:44:27.442486 | controller |     - '    logger.go:42: 09:44:01 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.442490 | controller |       1 == 1 '']'''
2026-03-14 09:44:27.442494 | controller |     - '    logger.go:42: 09:44:01 | watcher-api-scaling/1-deploy-with-defaults | ++ grep
2026-03-14 09:44:27.442499 | controller |       watcher'
2026-03-14 09:44:27.442503 | controller |     - '    logger.go:42: 09:44:01 | watcher-api-scaling/1-deploy-with-defaults | ++ awk
2026-03-14 09:44:27.442509 | controller |       ''{print $1}'''
2026-03-14 09:44:27.442514 | controller |     - '    logger.go:42: 09:44:01 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:44:27.442518 | controller |       exec -n watcher-kuttl-default openstackclient -- openstack service list -f value
2026-03-14 09:44:27.442522 | controller |       -c Name -c Type -c ID'
2026-03-14 09:44:27.442526 | controller |     - '    logger.go:42: 09:44:03 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=86b564fd237a41c2b773bfe0f05b75f1'
2026-03-14 09:44:27.442531 | controller |     - '    logger.go:42: 09:44:03 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:44:27.442535 | controller |       get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}'''
2026-03-14 09:44:27.442539 | controller |     - '    logger.go:42: 09:44:04 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.442543 | controller |       86b564fd237a41c2b773bfe0f05b75f1 == 86b564fd237a41c2b773bfe0f05b75f1 '']'''
2026-03-14 09:44:27.442547 | controller |     - '    logger.go:42: 09:44:04 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:44:27.442551 | controller |       get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}'''
2026-03-14 09:44:27.442556 | controller |     - '    logger.go:42: 09:44:04 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.442560 | controller |       -n n65fh675hf5h5f5h644h597h657h5d9h684h685h699h9fhd7h78h5d5h5c5h668h4h8bhc8h664h66fh6bh668h99h64fh564h5cbh5dfh8h7h5bdq
2026-03-14 09:44:27.442564 | controller |       '']'''
2026-03-14 09:44:27.442568 | controller |     - '    logger.go:42: 09:44:04 | watcher-api-scaling/1-deploy-with-defaults | ++ oc
2026-03-14 09:44:27.442572 | controller |       get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher'
2026-03-14 09:44:27.442577 | controller |     - '    logger.go:42: 09:44:04 | watcher-api-scaling/1-deploy-with-defaults | + ''[''
2026-03-14 09:44:27.442581 | controller |       '''' == '''' '']'''
2026-03-14 09:44:27.442585 | controller |     - '    logger.go:42: 09:44:04 | watcher-api-scaling/1-deploy-with-defaults | + exit
2026-03-14 09:44:27.442589 | controller |       0'
2026-03-14 09:44:27.442594 | controller |     - '    logger.go:42: 09:44:04 | watcher-api-scaling/1-deploy-with-defaults | test
2026-03-14 09:44:27.442598 | controller |       step completed 1-deploy-with-defaults'
2026-03-14 09:44:27.442602 | controller |     - '    logger.go:42: 09:44:04 | watcher-api-scaling/2-scale-up-watcher-api | starting
2026-03-14 09:44:27.442606 | controller |       test step 2-scale-up-watcher-api'
2026-03-14 09:44:27.442611 | controller |     - '    logger.go:42: 09:44:04 | watcher-api-scaling/2-scale-up-watcher-api | running
2026-03-14 09:44:27.442615 | controller |       command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op":
2026-03-14 09:44:27.442621 | controller |       "replace", "path": "/spec/apiServiceTemplate/replicas", "value":3}]'''
2026-03-14 09:44:27.442627 | controller |     - '        ]'
2026-03-14 09:44:27.442633 | controller |     - '    logger.go:42: 09:44:04 | watcher-api-scaling/2-scale-up-watcher-api | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:44:27.442639 | controller |       patched'
2026-03-14 09:44:27.442645 | controller |     - '    logger.go:42: 09:44:15 | watcher-api-scaling/2-scale-up-watcher-api | test
2026-03-14 09:44:27.442651 | controller |       step completed 2-scale-up-watcher-api'
2026-03-14 09:44:27.442655 | controller |     - '    logger.go:42: 09:44:15 | watcher-api-scaling/3-scale-down-watcher-api | starting
2026-03-14 09:44:27.442659 | controller |       test step 3-scale-down-watcher-api'
2026-03-14 09:44:27.442663 | controller |     - '    logger.go:42: 09:44:15 | watcher-api-scaling/3-scale-down-watcher-api | running
2026-03-14 09:44:27.442668 | controller |       command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op":
2026-03-14 09:44:27.442672 | controller |       "replace", "path": "/spec/apiServiceTemplate/replicas", "value":1}]'''
2026-03-14 09:44:27.442677 | controller |     - '        ]'
2026-03-14 09:44:27.442682 | controller |     - '    logger.go:42: 09:44:16 | watcher-api-scaling/3-scale-down-watcher-api | watcher.watcher.openstack.org/watcher-kuttl
2026-03-14 09:44:27.442688 | controller |       patched'
2026-03-14 09:44:27.442700 | controller |     - '    logger.go:42: 09:44:19 | watcher-api-scaling/3-scale-down-watcher-api | test
2026-03-14 09:44:27.442710 | controller |       step completed 3-scale-down-watcher-api'
2026-03-14 09:44:27.442716 | controller |     - '    logger.go:42: 09:44:19 | watcher-api-scaling/4-scale-down-zero-watcher-api
2026-03-14 09:44:27.442722 | controller |       | starting test step 4-scale-down-zero-watcher-api'
2026-03-14 09:44:27.442728 | controller |     - '    logger.go:42: 09:44:19 | watcher-api-scaling/4-scale-down-zero-watcher-api
2026-03-14 09:44:27.442733 | controller |       | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json''
2026-03-14 09:44:27.442738 | controller |       -p=''[{"op": "replace", "path": "/spec/apiServiceTemplate/replicas", "value":0}]'''
2026-03-14 09:44:27.442744 | controller |     - '        ]'
2026-03-14 09:44:27.442749 | controller |     - '    logger.go:42: 09:44:19 | watcher-api-scaling/4-scale-down-zero-watcher-api
2026-03-14 09:44:27.442755 | controller |       | watcher.watcher.openstack.org/watcher-kuttl patched'
2026-03-14 09:44:27.442761 | controller |     - '    logger.go:42: 09:44:20 | watcher-api-scaling/4-scale-down-zero-watcher-api
2026-03-14 09:44:27.442766 | controller |       | test step completed 4-scale-down-zero-watcher-api'
2026-03-14 09:44:27.442771 | controller |     - '    logger.go:42: 09:44:20 | watcher-api-scaling/5-cleanup-watcher | starting test
2026-03-14 09:44:27.442777 | controller |       step 5-cleanup-watcher'
2026-03-14 09:44:27.442783 | controller |     - '    logger.go:42: 09:44:20 | watcher-api-scaling/5-cleanup-watcher | running command:
2026-03-14 09:44:27.442788 | controller |       [sh -c set -ex'
2026-03-14 09:44:27.442794 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.442800 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:44:27.442806 | controller |     - '        ]'
2026-03-14 09:44:27.442811 | controller |     - '    logger.go:42: 09:44:20 | watcher-api-scaling/5-cleanup-watcher | + oc exec
2026-03-14 09:44:27.442817 | controller |       -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.442822 | controller |       -c Type'
2026-03-14 09:44:27.442828 | controller |     - '    logger.go:42: 09:44:20 | watcher-api-scaling/5-cleanup-watcher | ++ grep -c
2026-03-14 09:44:27.442834 | controller |       ''^watcher'''
2026-03-14 09:44:27.442841 | controller |     - '    logger.go:42: 09:44:22 | watcher-api-scaling/5-cleanup-watcher | + ''['' 1
2026-03-14 09:44:27.442846 | controller |       == 0 '']'''
2026-03-14 09:44:27.442852 | controller |     - '    logger.go:42: 09:44:24 | watcher-api-scaling/5-cleanup-watcher | running command:
2026-03-14 09:44:27.442858 | controller |       [sh -c set -ex'
2026-03-14 09:44:27.442863 | controller |     - '        oc exec -n watcher-kuttl-default openstackclient -- openstack service list
2026-03-14 09:44:27.442869 | controller |       -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]'
2026-03-14 09:44:27.442875 | controller |     - '        ]'
2026-03-14 09:44:27.442881 | controller |     - '    logger.go:42: 09:44:24 | watcher-api-scaling/5-cleanup-watcher | + oc exec
2026-03-14 09:44:27.442886 | controller |       -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name
2026-03-14 09:44:27.442892 | controller |       -c Type'
2026-03-14 09:44:27.442898 | controller |     - '    logger.go:42: 09:44:24 | watcher-api-scaling/5-cleanup-watcher | ++ grep -c
2026-03-14 09:44:27.442903 | controller |       ''^watcher'''
2026-03-14 09:44:27.442913 | controller |     - '    logger.go:42: 09:44:26 | watcher-api-scaling/5-cleanup-watcher | + ''['' 0
2026-03-14 09:44:27.442918 | controller |       == 0 '']'''
2026-03-14 09:44:27.442925 | controller |     - '    logger.go:42: 09:44:26 | watcher-api-scaling/5-cleanup-watcher | test step
2026-03-14 09:44:27.442932 | controller |       completed 5-cleanup-watcher'
2026-03-14 09:44:27.442938 | controller |     - '    logger.go:42: 09:44:26 | watcher-api-scaling | skipping kubernetes event logging'
2026-03-14 09:44:27.442945 | controller |     - === CONT  kuttl/harness/deps
2026-03-14 09:44:27.442952 | controller |     - '    logger.go:42: 09:44:26 | deps | Ignoring infra.yaml as it does not match file
2026-03-14 09:44:27.442957 | controller |       name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$'
2026-03-14 09:44:27.442963 | controller |     - '    logger.go:42: 09:44:26 | deps | Ignoring keystone.yaml as it does not match
2026-03-14 09:44:27.442971 | controller |       file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$'
2026-03-14 09:44:27.442977 | controller |     - '    logger.go:42: 09:44:26 | deps | Ignoring kustomization.yaml as it does not
2026-03-14 09:44:27.442982 | controller |       match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$'
2026-03-14 09:44:27.442988 | controller |     - '    logger.go:42: 09:44:26 | deps | Ignoring namespace.yaml as it does not match
2026-03-14 09:44:27.442994 | controller |       file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$'
2026-03-14 09:44:27.443000 | controller |     - '    logger.go:42: 09:44:26 | deps | Ignoring telemetry.yaml as it does not match
2026-03-14 09:44:27.443006 | controller |       file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$'
2026-03-14 09:44:27.443011 | controller |     - '    logger.go:42: 09:44:26 | deps | Skipping creation of user-supplied namespace:
2026-03-14 09:44:27.443017 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.443022 | controller |     - '    logger.go:42: 09:44:26 | deps | skipping kubernetes event logging'
2026-03-14 09:44:27.443028 | controller |     - === CONT  kuttl/harness/common
2026-03-14 09:44:27.443033 | controller |     - '    logger.go:42: 09:44:26 | common | Ignoring cleanup-assert.yaml as it does not
2026-03-14 09:44:27.443039 | controller |       match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$'
2026-03-14 09:44:27.443044 | controller |     - '    logger.go:42: 09:44:26 | common | Ignoring cleanup-errors.yaml as it does not
2026-03-14 09:44:27.443050 | controller |       match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$'
2026-03-14 09:44:27.443056 | controller |     - '    logger.go:42: 09:44:26 | common | Ignoring cleanup-watcher.yaml as it does
2026-03-14 09:44:27.443062 | controller |       not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$'
2026-03-14 09:44:27.443070 | controller |     - '    logger.go:42: 09:44:26 | common | Ignoring deploy-with-defaults.yaml as it
2026-03-14 09:44:27.443076 | controller |       does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$'
2026-03-14 09:44:27.443086 | controller |     - '    logger.go:42: 09:44:26 | common | Skipping creation of user-supplied namespace:
2026-03-14 09:44:27.528628 | controller |       watcher-kuttl-default'
2026-03-14 09:44:27.528683 | controller |     - '    logger.go:42: 09:44:26 | common | skipping kubernetes event logging'
2026-03-14 09:44:27.528690 | controller |     - === NAME  kuttl
2026-03-14 09:44:27.528695 | controller |     - '    harness.go:406: run tests finished'
2026-03-14 09:44:27.528700 | controller |     - '    harness.go:514: cleaning up'
2026-03-14 09:44:27.528704 | controller |     - '    harness.go:571: removing temp folder: ""'
2026-03-14 09:44:27.528709 | controller |     - '--- FAIL: kuttl (1507.06s)'
2026-03-14 09:44:27.528713 | controller |     - '    --- FAIL: kuttl/harness (0.00s)'
2026-03-14 09:44:27.528717 | controller |     - '        --- FAIL: kuttl/harness/appcred-tests (951.33s)'
2026-03-14 09:44:27.528722 | controller |     - '        --- PASS: kuttl/harness/watcher-notification (40.72s)'
2026-03-14 09:44:27.528731 | controller |     - '        --- PASS: kuttl/harness/watcher-topology (33.46s)'
2026-03-14 09:44:27.528736 | controller |     - '        --- PASS: kuttl/harness/watcher-tls-certs-change (35.80s)'
2026-03-14 09:44:27.528740 | controller |     - '        --- PASS: kuttl/harness/watcher-tls (134.34s)'
2026-03-14 09:44:27.528746 | controller |     - '        --- PASS: kuttl/harness/watcher-rmquser (34.76s)'
2026-03-14 09:44:27.528752 | controller |     - '        --- PASS: kuttl/harness/watcher (88.99s)'
2026-03-14 09:44:27.528757 | controller |     - '        --- PASS: kuttl/harness/watcher-cinder (131.87s)'
2026-03-14 09:44:27.528762 | controller |     - '        --- PASS: kuttl/harness/watcher-api-scaling (55.74s)'
2026-03-14 09:44:27.528768 | controller |     - '        --- PASS: kuttl/harness/deps (0.00s)'
2026-03-14 09:44:27.528774 | controller |     - '        --- PASS: kuttl/harness/common (0.02s)'
2026-03-14 09:44:27.528779 | controller |     - FAIL
2026-03-14 09:44:27.528794 | controller |
2026-03-14 09:44:27.602739 | controller | NO MORE HOSTS LEFT *************************************************************
2026-03-14 09:44:27.602868 | controller |
2026-03-14 09:44:27.602877 | controller | PLAY RECAP *********************************************************************
2026-03-14 09:44:27.602882 | controller | controller                 : ok=2    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0
2026-03-14 09:44:27.602898 | controller |
2026-03-14 09:44:27.602902 | controller | Saturday 14 March 2026  09:44:27 +0000 (0:25:08.820)       0:25:09.223 ********
2026-03-14 09:44:27.602907 | controller | ===============================================================================
2026-03-14 09:44:27.602911 | controller | run kuttl test suite from operator Makefile -------------------------- 1508.82s
2026-03-14 09:44:27.602916 | controller | run_hook : Loop on hooks for pre_kuttl_from_operator -------------------- 0.15s
2026-03-14 09:44:27.602920 | controller | run_hook : Assert single hooks are all mappings ------------------------- 0.11s
2026-03-14 09:44:27.602925 | controller | run_hook : Assert parameters are valid ---------------------------------- 0.07s
2026-03-14 09:44:27.602930 | controller | Run hooks before running kuttl tests ------------------------------------ 0.03s
2026-03-14 09:44:27.602936 | controller | Saturday 14 March 2026  09:44:27 +0000 (0:25:08.820)       0:25:09.222 ********
2026-03-14 09:44:27.602941 | controller | ===============================================================================
2026-03-14 09:44:27.602945 | controller | ansible.builtin.command ---------------------------------------------- 1508.82s
2026-03-14 09:44:27.602951 | controller | run_hook ---------------------------------------------------------------- 0.32s
2026-03-14 09:44:27.602956 | controller | ansible.builtin.include_role -------------------------------------------- 0.03s
2026-03-14 09:44:27.602961 | controller | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2026-03-14 09:44:27.602965 | controller | total ---------------------------------------------------------------- 1509.18s
2026-03-14 09:44:28.760600 | controller | ERROR
2026-03-14 09:44:28.761062 | controller | {
2026-03-14 09:44:28.761135 | controller |   "delta": "0:25:09.880086",
2026-03-14 09:44:28.761182 | controller |   "end": "2026-03-14 09:44:27.608517",
2026-03-14 09:44:28.761222 | controller |   "msg": "non-zero return code",
2026-03-14 09:44:28.761260 | controller |   "rc": 2,
2026-03-14 09:44:28.761296 | controller |   "start": "2026-03-14 09:19:17.728431"
2026-03-14 09:44:28.761331 | controller | }
failure
2026-03-14 09:44:29.491323 | 
2026-03-14 09:44:29.491944 | PLAY RECAP
2026-03-14 09:44:29.492085 | controller | ok: 0 changed: 0 unreachable: 0 failed: 1 skipped: 0 rescued: 0 ignored: 0
2026-03-14 09:44:29.492143 | 
2026-03-14 09:44:29.742802 | RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/kuttl-from-operator-run.yaml@main]
2026-03-14 09:44:29.747688 | POST-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/e2e-collect-logs.yml@main]
2026-03-14 09:44:30.452333 | 
2026-03-14 09:44:30.452565 | PLAY [Run ci/playbooks/e2e-collect-logs.yml]
2026-03-14 09:44:30.500648 | 
2026-03-14 09:44:30.500843 | TASK [Filter out host if needed]
2026-03-14 09:44:30.512983 | controller | skipping: end_host conditional evaluated to False, continuing execution for controller
2026-03-14 09:44:30.519309 | 
2026-03-14 09:44:30.519391 | TASK [Filter out host if needed]
2026-03-14 09:44:30.535185 | 
2026-03-14 09:44:30.535321 | TASK [Ensure we have the ci-framework on host]
2026-03-14 09:44:31.127232 | controller | ok
2026-03-14 09:44:31.145763 | 
2026-03-14 09:44:31.145950 | TASK [End host if no basedir]
2026-03-14 09:44:31.162214 | controller | skipping: end_host conditional evaluated to False, continuing execution for controller
2026-03-14 09:44:31.172268 | 
2026-03-14 09:44:31.172373 | TASK [Read base centos-9 scenarios]
2026-03-14 09:44:31.197639 | controller | ok
2026-03-14 09:44:31.309637 | 
2026-03-14 09:44:31.309799 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension]
2026-03-14 09:44:31.344583 | controller | skipping: Conditional result was False
2026-03-14 09:44:31.353874 | 
2026-03-14 09:44:31.353965 | TASK [cifmw_helpers : Check if file is available]
2026-03-14 09:44:31.619862 | controller | ok
2026-03-14 09:44:31.625978 | 
2026-03-14 09:44:31.626062 | TASK [cifmw_helpers : Create temporary directory]
2026-03-14 09:44:31.671574 | controller | skipping: Conditional result was False
2026-03-14 09:44:31.679485 | 
2026-03-14 09:44:31.679569 | TASK [cifmw_helpers : Read vars]
2026-03-14 09:44:31.714113 | 
2026-03-14 09:44:31.714346 | TASK [cifmw_helpers : Create new variable files with content]
2026-03-14 09:44:31.760520 | 
2026-03-14 09:44:31.760742 | TASK [cifmw_helpers : Include vars]
2026-03-14 09:44:31.807165 | 
2026-03-14 09:44:31.807423 | TASK [cifmw_helpers : Remove temporary directory]
2026-03-14 09:44:31.854284 | controller | skipping: Conditional result was False
2026-03-14 09:44:31.868099 | 
2026-03-14 09:44:31.868182 | TASK [run_hook : Assert parameters are valid]
2026-03-14 09:44:31.940095 | controller | ok: All assertions passed
2026-03-14 09:44:31.946302 | 
2026-03-14 09:44:31.946366 | TASK [run_hook : Assert single hooks are all mappings]
2026-03-14 09:44:32.029798 | controller | ok: All assertions passed
2026-03-14 09:44:32.038459 | 
2026-03-14 09:44:32.038554 | LOOP [run_hook : Loop on hooks for pre_logs]
2026-03-14 09:44:32.117141 | 
2026-03-14 09:44:32.117336 | TASK [cifmw_setup : Ensure cifmw_basedir param is set]
2026-03-14 09:44:32.142397 | controller | skipping: Conditional result was False
2026-03-14 09:44:32.150737 | 
2026-03-14 09:44:32.150827 | TASK [Read artifacts parameters dir and set as facts]
2026-03-14 09:44:32.175120 | controller | ok
2026-03-14 09:44:32.200585 | 
2026-03-14 09:44:32.200751 | TASK [cifmw_helpers : Check directory is available]
2026-03-14 09:44:32.448315 | controller | ok
2026-03-14 09:44:32.457167 | 
2026-03-14 09:44:32.457264 | TASK [cifmw_helpers : Find yaml files]
2026-03-14 09:44:32.788759 | controller | ok: All paths examined
2026-03-14 09:44:32.799690 | 
2026-03-14 09:44:32.799809 | TASK [cifmw_helpers : Print available yaml files]
2026-03-14 09:44:32.864299 | controller | ok: Found yaml files to parse: ['/home/zuul/ci-framework-data/artifacts/parameters/zuul-params.yml', '/home/zuul/ci-framework-data/artifacts/parameters/install-yamls-params.yml', '/home/zuul/ci-framework-data/artifacts/parameters/custom-params.yml', '/home/zuul/ci-framework-data/artifacts/parameters/openshift-login-params.yml']
2026-03-14 09:44:32.883292 | 
2026-03-14 09:44:32.883456 | LOOP [cifmw_helpers : Create files on localhost and use include_vars]
2026-03-14 09:44:32.937450 | controller | Output suppressed because no_log was given
2026-03-14 09:44:32.938480 | 
2026-03-14 09:44:32.954372 | controller | Output suppressed because no_log was given
2026-03-14 09:44:32.967306 | controller | Output suppressed because no_log was given
2026-03-14 09:44:32.979809 | controller | Output suppressed because no_log was given
2026-03-14 09:44:33.009505 | controller | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/cifmw_helpers/tasks/include_file.yml
2026-03-14 09:44:33.015843 | controller | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/cifmw_helpers/tasks/include_file.yml
2026-03-14 09:44:33.022256 | controller | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/cifmw_helpers/tasks/include_file.yml
2026-03-14 09:44:33.028984 | controller | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/cifmw_helpers/tasks/include_file.yml
2026-03-14 09:44:33.040691 | 
2026-03-14 09:44:33.040774 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension]
2026-03-14 09:44:33.077157 | controller | skipping: Conditional result was False
2026-03-14 09:44:33.087224 | 
2026-03-14 09:44:33.087313 | TASK [cifmw_helpers : Check if file is available]
2026-03-14 09:44:33.351286 | controller | ok
2026-03-14 09:44:33.365900 | 
2026-03-14 09:44:33.366063 | TASK [cifmw_helpers : Create temporary directory]
2026-03-14 09:44:33.802358 | controller -> localhost | changed
2026-03-14 09:44:33.819324 | 
2026-03-14 09:44:33.819476 | TASK [cifmw_helpers : Read vars]
2026-03-14 09:44:34.228724 | controller | Output suppressed because no_log was given
2026-03-14 09:44:34.244734 | 
2026-03-14 09:44:34.244908 | TASK [cifmw_helpers : Create new variable files with content]
2026-03-14 09:44:34.937407 | controller -> localhost | Output suppressed because no_log was given
2026-03-14 09:44:34.945133 | 
2026-03-14 09:44:34.945210 | TASK [cifmw_helpers : Include vars]
2026-03-14 09:44:34.991707 | controller -> localhost | Output suppressed because no_log was given
2026-03-14 09:44:34.999416 | 
2026-03-14 09:44:34.999501 | TASK [cifmw_helpers : Remove temporary directory]
2026-03-14 09:44:35.475994 | controller -> localhost | changed
2026-03-14 09:44:35.492859 | 
2026-03-14 09:44:35.493066 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension]
2026-03-14 09:44:35.531640 | controller | skipping: Conditional result was False
2026-03-14 09:44:35.547380 | 
2026-03-14 09:44:35.547550 | TASK [cifmw_helpers : Check if file is available]
2026-03-14 09:44:35.856383 | controller | ok
2026-03-14 09:44:35.874779 | 
2026-03-14 09:44:35.874958 | TASK [cifmw_helpers : Create temporary directory]
2026-03-14 09:44:36.190114 | controller -> localhost | changed
2026-03-14 09:44:36.206061 | 
2026-03-14 09:44:36.206213 | TASK [cifmw_helpers : Read vars]
2026-03-14 09:44:36.451107 | controller | Output suppressed because no_log was given
2026-03-14 09:44:36.464657 | 
2026-03-14 09:44:36.464788 | TASK [cifmw_helpers : Create new variable files with content]
2026-03-14 09:44:37.013351 | controller -> localhost | Output suppressed because no_log was given
2026-03-14 09:44:37.032427 | 
2026-03-14 09:44:37.032585 | TASK [cifmw_helpers : Include vars]
2026-03-14 09:44:37.105407 | controller -> localhost | Output suppressed because no_log was given
2026-03-14 09:44:37.124295 | 
2026-03-14 09:44:37.124450 | TASK [cifmw_helpers : Remove temporary directory]
2026-03-14 09:44:37.431575 | controller -> localhost | changed
2026-03-14 09:44:37.447008 | 
2026-03-14 09:44:37.447508 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension]
2026-03-14 09:44:37.485871 | controller | skipping: Conditional result was False
2026-03-14 09:44:37.504301 | 
2026-03-14 09:44:37.504489 | TASK [cifmw_helpers : Check if file is available]
2026-03-14 09:44:37.874376 | controller | ok
2026-03-14 09:44:37.888586 | 
2026-03-14 09:44:37.888736 | TASK [cifmw_helpers : Create temporary directory]
2026-03-14 09:44:38.182591 | controller -> localhost | changed
2026-03-14 09:44:38.195941 | 
2026-03-14 09:44:38.196124 | TASK [cifmw_helpers : Read vars]
2026-03-14 09:44:38.416073 | controller | Output suppressed because no_log was given
2026-03-14 09:44:38.425232 | 
2026-03-14 09:44:38.425323 | TASK [cifmw_helpers : Create new variable files with content]
2026-03-14 09:44:38.913495 | controller -> localhost | Output suppressed because no_log was given
2026-03-14 09:44:38.929749 | 
2026-03-14 09:44:38.929969 | TASK [cifmw_helpers : Include vars]
2026-03-14 09:44:38.990642 | controller -> localhost | Output suppressed because no_log was given
2026-03-14 09:44:39.009854 | 
2026-03-14 09:44:39.010151 | TASK [cifmw_helpers : Remove temporary directory]
2026-03-14 09:44:39.316412 | controller -> localhost | changed
2026-03-14 09:44:39.332853 | 
2026-03-14 09:44:39.333040 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension]
2026-03-14 09:44:39.371970 | controller | skipping: Conditional result was False
2026-03-14 09:44:39.394390 | 
2026-03-14 09:44:39.394647 | TASK [cifmw_helpers : Check if file is available]
2026-03-14 09:44:39.662997 | controller | ok
2026-03-14 09:44:39.678541 | 
2026-03-14 09:44:39.678683 | TASK [cifmw_helpers : Create temporary directory]
2026-03-14 09:44:39.964575 | controller -> localhost | changed
2026-03-14 09:44:39.980326 | 
2026-03-14 09:44:39.980477 | TASK [cifmw_helpers : Read vars]
2026-03-14 09:44:40.210260 | controller | Output suppressed because no_log was given
2026-03-14 09:44:40.227459 | 
2026-03-14 09:44:40.227613 | TASK [cifmw_helpers : Create new variable files with content]
2026-03-14 09:44:40.764741 | controller -> localhost | Output suppressed because no_log was given
2026-03-14 09:44:40.772885 | 
2026-03-14 09:44:40.772954 | TASK [cifmw_helpers : Include vars]
2026-03-14 09:44:40.808732 | controller -> localhost | Output suppressed because no_log was given
2026-03-14 09:44:40.830806 | 
2026-03-14 09:44:40.831005 | TASK [cifmw_helpers : Remove temporary directory]
2026-03-14 09:44:41.136178 | controller -> localhost | changed
2026-03-14 09:44:41.151741 | 
2026-03-14 09:44:41.151850 | TASK [cifmw_setup : Set custom cifmw PATH reusable fact]
2026-03-14 09:44:41.188174 | controller | skipping: Conditional result was False
2026-03-14 09:44:41.198449 | 
2026-03-14 09:44:41.198545 | TASK [cifmw_setup : Set destination folder for the logs]
2026-03-14 09:44:41.249188 | controller | ok
2026-03-14 09:44:41.260500 | 
2026-03-14 09:44:41.260588 | TASK [artifacts : Verify if "success" flag exists after successful tests execution]
2026-03-14 09:44:41.474876 | controller | ok
2026-03-14 09:44:41.489155 | 
2026-03-14 09:44:41.489298 | TASK [artifacts : Ensure we have at least minimal ansible facts]
2026-03-14 09:44:42.149363 | controller | ok
2026-03-14 09:44:42.165341 | 
2026-03-14 09:44:42.165479 | LOOP [artifacts : Ensure base directory exists]
2026-03-14 09:44:42.426241 | controller | ok: "artifacts"
2026-03-14 09:44:42.603208 | controller | ok: "logs"
2026-03-14 09:44:42.626169 | 
2026-03-14 09:44:42.626389 | TASK [artifacts : Load generated hook environment for further usage]
2026-03-14 09:44:42.667798 | controller | ERROR
2026-03-14 09:44:42.668269 | controller | {
2026-03-14 09:44:42.668330 | controller |   "ansible_facts": {},
2026-03-14 09:44:42.668370 | controller |   "ansible_included_var_files": [],
2026-03-14 09:44:42.668406 | controller |   "message": "/home/zuul/ci-framework-data/artifacts directory does not exist"
2026-03-14 09:44:42.668443 | controller | }
2026-03-14 09:44:42.668490 | controller | ERROR: Ignoring Errors
2026-03-14 09:44:42.681498 | 
2026-03-14 09:44:42.681649 | TASK [artifacts : Gather parameter passed to the playbook]
2026-03-14 09:44:43.405823 | controller | changed
2026-03-14 09:44:43.419301 | 
2026-03-14 09:44:43.419449 | TASK [artifacts : Gather facts]
2026-03-14 09:44:44.081051 | controller | changed
2026-03-14 09:44:44.101222 | 
2026-03-14 09:44:44.101452 | TASK [artifacts : Copy some network configuration files]
2026-03-14 09:44:54.715683 | controller | changed
2026-03-14 09:44:54.722796 | 
2026-03-14 09:44:54.722860 | TASK [artifacts : Get installed packages list]
2026-03-14 09:44:55.879936 | controller | ok
2026-03-14 09:44:55.943061 | 
2026-03-14 09:44:55.943220 | TASK [artifacts : Output package list]
2026-03-14 09:44:56.772069 | controller | changed
2026-03-14 09:44:56.802990 | 
2026-03-14 09:44:56.803188 | LOOP [artifacts : Cleanup unnecesary directories]
2026-03-14 09:44:57.124411 | controller | changed: "/home/zuul/ci-framework-data/artifacts/manifests/operator"
2026-03-14 09:44:57.124793 | controller | ok: All items complete
2026-03-14 09:44:57.124836 | 
2026-03-14 09:44:57.180686 | 
2026-03-14 09:44:57.180895 | TASK [os_must_gather : Ensure directories are present]
2026-03-14 09:44:57.450903 | controller | changed
2026-03-14 09:44:57.478640 | 
2026-03-14 09:44:57.478842 | LOOP [os_must_gather : Construct project change list]
2026-03-14 09:44:57.563684 | controller | ok:
2026-03-14 09:44:57.564051 | controller | {
2026-03-14 09:44:57.564111 | controller |   "branch": "18.0-fr5",
2026-03-14 09:44:57.564141 | controller |   "change": "353",
2026-03-14 09:44:57.564165 | controller |   "change_message": "openstack-k8s-operators dependency bump branch: 18.0-fr5\n\nAutomated changes by [create-pull-request](https://github.com/peter-evans/create-pull-request) GitHub action",
2026-03-14 09:44:57.564195 | controller |   "change_url": "https://github.com/openstack-k8s-operators/watcher-operator/pull/353",
2026-03-14 09:44:57.564219 | controller |   "commit_id": "3fad4a9eb56718f26ce2ec186bb570f2695f01c3",
2026-03-14 09:44:57.564243 | controller |   "patchset": "3fad4a9eb56718f26ce2ec186bb570f2695f01c3",
2026-03-14 09:44:57.564265 | controller |   "project": {
2026-03-14 09:44:57.564288 | controller |     "canonical_hostname": "github.com",
2026-03-14 09:44:57.564311 | controller |     "canonical_name": "github.com/openstack-k8s-operators/watcher-operator",
2026-03-14 09:44:57.564333 | controller |     "name": "openstack-k8s-operators/watcher-operator",
2026-03-14 09:44:57.564355 | controller |     "short_name": "watcher-operator",
2026-03-14 09:44:57.564377 | controller |     "src_dir": "src/github.com/openstack-k8s-operators/watcher-operator"
2026-03-14 09:44:57.564399 | controller |   },
2026-03-14 09:44:57.564420 | controller |   "topic": null
2026-03-14 09:44:57.564442 | controller | }
2026-03-14 09:44:57.631010 | 
2026-03-14 09:44:57.631247 | TASK [os_must_gather : Get git tag for image tagging]
2026-03-14 09:44:58.314664 | controller | skipping: Conditional result was False
2026-03-14 09:44:58.360186 | 
2026-03-14 09:44:58.360420 | TASK [os_must_gather : Debug git show-ref]
2026-03-14 09:44:58.387835 | controller | skipping: Conditional result was False
2026-03-14 09:44:58.422716 | 
2026-03-14 09:44:58.422894 | TASK [os_must_gather : Build openstack-must-gather container]
2026-03-14 09:44:58.449985 | controller | skipping: Conditional result was False
2026-03-14 09:44:58.477734 | 
2026-03-14 09:44:58.477884 | TASK [os_must_gather : Push openstack-must-gather container]
2026-03-14 09:44:58.513776 | controller | skipping: Conditional result was False
2026-03-14 09:44:58.557648 | 
2026-03-14 09:44:58.557797 | TASK [os_must_gather : Set cifmw_os_must_gather_image to new image tag]
2026-03-14 09:44:58.594974 | controller | skipping: Conditional result was False
2026-03-14 09:44:58.628627 | 
2026-03-14 09:44:58.628787 | TASK [os_must_gather : Check for oc command]
2026-03-14 09:44:59.017350 | controller | changed
2026-03-14 09:44:59.040542 | 
2026-03-14 09:44:59.040611 | TASK [os_must_gather : Check if kubeconfig exists]
2026-03-14 09:44:59.323595 | controller | ok
2026-03-14 09:44:59.421138 | 
2026-03-14 09:44:59.421323 | TASK [os_must_gather : Run openstack-must-gather command]
2026-03-14 09:49:53.514090 | controller | changed
2026-03-14 09:49:53.557872 | 
2026-03-14 09:49:53.557999 | TASK [os_must_gather : Find existing os-must-gather directories]
2026-03-14 09:49:53.847101 | controller | ok: All paths examined
2026-03-14 09:49:53.877546 | 
2026-03-14 09:49:53.877719 | TASK [os_must_gather : Create a symlink to newest os-must-gather directory]
2026-03-14 09:49:54.220490 | controller | changed
2026-03-14 09:49:54.237877 | 
2026-03-14 09:49:54.237946 | TASK [os_must_gather : Collect pod usage]
2026-03-14 09:49:54.291532 | controller | ok
2026-03-14 09:49:54.313593 | controller | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/os_must_gather/tasks/get_top.yml
2026-03-14 09:49:54.335194 | 
2026-03-14 09:49:54.335270 | TASK [os_must_gather : Get all containers usage - sort by cpu]
2026-03-14 09:49:54.845933 | controller | error: Metrics API not available
2026-03-14 09:50:04.975983 | controller | ERROR
2026-03-14 09:50:04.976359 | controller | {
2026-03-14 09:50:04.976479 | controller |   "delta": "0:00:00.164666",
2026-03-14 09:50:04.976524 | controller |   "end": "2026-03-14 09:49:54.849231",
2026-03-14 09:50:04.976565 | controller |   "msg": "non-zero return code",
2026-03-14 09:50:04.976603 | controller |   "rc": 1,
2026-03-14 09:50:04.976645 | controller |   "start": "2026-03-14 09:49:54.684565"
2026-03-14 09:50:04.976688 | controller | }
2026-03-14 09:50:04.976737 | controller | ERROR: Ignoring Errors
2026-03-14 09:50:05.016739 | 
2026-03-14 09:50:05.016900 | TASK [os_must_gather : Print message when metrics are not available]
2026-03-14 09:50:05.065891 | controller | skipping: Conditional result was False
2026-03-14 09:50:05.110370 | 
2026-03-14 09:50:05.110590 | TASK [os_must_gather : Get all containers usage - sort by memory]
2026-03-14 09:50:05.792946 | controller | skipping: Conditional result was False
2026-03-14 09:50:05.835836 | 
2026-03-14 09:50:05.836061 | TASK [os_must_gather : Find all namespaces directories]
2026-03-14 09:50:05.874607 | controller | skipping: Conditional result was False
2026-03-14 09:50:05.903057 | 
2026-03-14 09:50:05.903174 | LOOP [os_must_gather : Get resource usage by pods per namespace]
2026-03-14 09:50:05.985235 | 
2026-03-14 09:50:05.985535 | TASK [os_must_gather : Get node resource usage]
2026-03-14 09:50:06.659901 | controller | skipping: Conditional result was False
2026-03-14 09:50:06.729952 | 
2026-03-14 09:50:06.730102 | TASK [artifacts : Create crc logs directory]
2026-03-14 09:50:07.001292 | controller | changed
2026-03-14 09:50:07.029674 | 
2026-03-14 09:50:07.029781 | TASK [artifacts : Ensure controller knows CRC ssh keys]
2026-03-14 09:50:17.680914 | controller | changed
2026-03-14 09:50:17.723077 | 
2026-03-14 09:50:17.723284 | TASK [artifacts : Recognize new keypair]
2026-03-14 09:50:18.046773 | controller | ok
2026-03-14 09:50:18.064330 | 
2026-03-14 09:50:18.064404 | TASK [artifacts : Set fact if new keypair exists]
2026-03-14 09:50:18.109935 | controller | skipping: Conditional result was False
2026-03-14 09:50:18.129934 | 
2026-03-14 09:50:18.130040 | TASK [artifacts : Prepare root ssh accesses]
2026-03-14 09:50:22.880644 | controller | changed
2026-03-14 09:50:22.907812 | 
2026-03-14 09:50:22.907941 | TASK [artifacts : Copy logs from CRC VM]
2026-03-14 09:50:25.961895 | controller | changed
2026-03-14 09:50:25.994744 | 
2026-03-14 09:50:25.994845 | TASK [artifacts : List all of the existing virtual machines]
2026-03-14 09:50:26.031796 | controller | skipping: Conditional result was False
2026-03-14 09:50:26.071457 | 
2026-03-14 09:50:26.071564 | TASK [artifacts : Filter out edpm vm]
2026-03-14 09:50:26.107991 | controller | skipping: Conditional result was False
2026-03-14 09:50:26.135879 | 
2026-03-14 09:50:26.135984 | TASK [artifacts : Slurp zuul inventory file]
2026-03-14 09:50:26.381834 | controller | ok
2026-03-14 09:50:26.407969 | 
2026-03-14 09:50:26.408091 | LOOP [artifacts : Extract Compute and Networker from zuul mapping if any]
2026-03-14 09:50:26.632553 | 
2026-03-14 09:50:26.632716 | LOOP [artifacts : Generate logs on edpm vm {{ item.split('/')[0] }}]
2026-03-14 09:50:26.708915 | 
2026-03-14 09:50:26.709158 | LOOP [artifacts : Copy logs to host machine from {{ item.split('/')[0] }}]
2026-03-14 09:50:26.785814 | 
2026-03-14 09:50:26.786051 | TASK [artifacts : Generate list of ansible logs to collect in home directory]
2026-03-14 09:50:27.044886 | controller | ok: All paths examined
2026-03-14 09:50:27.069967 | 
2026-03-14 09:50:27.070120 | LOOP [artifacts : Copy ansible logs to logs directory]
2026-03-14 09:50:27.511511 | controller | changed:
2026-03-14 09:50:27.512071 | controller | {
2026-03-14 09:50:27.512147 | controller |   "atime": 1773479134.2584317,
2026-03-14 09:50:27.512189 | controller |   "ctime": 1773481467.527609,
2026-03-14 09:50:27.512225 | controller |   "dev": 64513,
2026-03-14 09:50:27.512259 | controller |   "gid": 1000,
2026-03-14 09:50:27.512292 | controller |   "gr_name": "zuul",
2026-03-14 09:50:27.512326 | controller |   "inode": 1247249,
2026-03-14 09:50:27.512358 | controller |   "isblk": false,
2026-03-14 09:50:27.512390 | controller |   "ischr": false,
2026-03-14 09:50:27.512422 | controller |   "isdir": false,
2026-03-14 09:50:27.512454 | controller |   "isfifo": false,
2026-03-14 09:50:27.512486 | controller |   "isgid": false,
2026-03-14 09:50:27.512518 | controller |   "islnk": false,
2026-03-14 09:50:27.512557 | controller |   "isreg": true,
2026-03-14 09:50:27.512592 | controller |   "issock": false,
2026-03-14 09:50:27.512625 | controller |   "isuid": false,
2026-03-14 09:50:27.512658 | controller |   "mode": "0644",
2026-03-14 09:50:27.512689 | controller |   "mtime": 1773481467.527609,
2026-03-14 09:50:27.512722 | controller |   "nlink": 1,
2026-03-14 09:50:27.512754 | controller |   "path": "/home/zuul/ansible.log",
2026-03-14 09:50:27.512786 | controller |   "pw_name": "zuul",
2026-03-14 09:50:27.512819 | controller |   "rgrp": true,
2026-03-14 09:50:27.512850 | controller |   "roth": true,
2026-03-14 09:50:27.512884 | controller |   "rusr": true,
2026-03-14 09:50:27.512918 | controller |   "size": 6103908,
2026-03-14 09:50:27.512951 | controller |   "uid": 1000,
2026-03-14 09:50:27.512984 | controller |   "wgrp": false,
2026-03-14 09:50:27.513046 | controller |   "woth": false,
2026-03-14 09:50:27.513092 | controller |   "wusr": true,
2026-03-14 09:50:27.513127 | controller |   "xgrp": false,
2026-03-14 09:50:27.513160 | controller |   "xoth": false,
2026-03-14 09:50:27.513193 | controller |   "xusr": false
2026-03-14 09:50:27.513226 | controller | }
2026-03-14 09:50:27.513282 | 
2026-03-14 09:50:27.560939 | 
2026-03-14 09:50:27.561225 | TASK [artifacts : Ensure we have proper rights on the gathered content]
2026-03-14 09:50:40.207221 | controller | changed
2026-03-14 09:50:40.238271 | 
2026-03-14 09:50:40.238483 | LOOP [artifacts : Mask secrets in yaml log files]
2026-03-14 09:53:34.956996 | controller | changed: "/home/zuul/ci-framework-data/logs"
2026-03-14 09:53:35.648966 | controller | changed: "/home/zuul/ci-framework-data/artifacts"
2026-03-14 09:53:35.691495 | 
2026-03-14 09:53:35.691650 | LOOP [env_op_images : Ensure directory is present]
2026-03-14 09:53:35.963777 | controller | ok: "artifacts"
2026-03-14 09:53:35.964247 | controller | changed: All items complete
2026-03-14 09:53:35.964307 | 
2026-03-14 09:53:36.153051 | controller | ok: "logs"
2026-03-14 09:53:36.202424 | 
2026-03-14 09:53:36.202661 | TASK [env_op_images : Check if OpenStackControlPlane is setup]
2026-03-14 09:53:46.905292 | controller | changed
2026-03-14 09:53:46.938679 | 
2026-03-14 09:53:46.938791 | TASK [env_op_images : Get images from the CSV]
2026-03-14 09:53:47.562813 | controller | skipping: Conditional result was False
2026-03-14 09:53:47.647376 | 
2026-03-14 09:53:47.647529 | TASK [env_op_images : Get the images name]
2026-03-14 09:53:48.272852 | controller | skipping: Conditional result was False
2026-03-14 09:53:48.301123 | 
2026-03-14 09:53:48.301246 | TASK [env_op_images : Extract env variable name and images]
2026-03-14 09:53:48.337162 | controller | skipping: Conditional result was False
2026-03-14 09:53:48.369419 | 
2026-03-14 09:53:48.369554 | TASK [env_op_images : Get all pods from all namespaces to find openstack-operator-index]
2026-03-14 09:53:48.406346 | controller | skipping: Conditional result was False
2026-03-14 09:53:48.446217 | 
2026-03-14 09:53:48.446383 | TASK [env_op_images : Retrieve openstack-operator-index pod]
2026-03-14 09:53:48.483432 | controller | skipping: Conditional result was False
2026-03-14 09:53:48.512464 | 
2026-03-14 09:53:48.512606 | TASK [env_op_images : Get all the pods in openstack-operator namespace]
2026-03-14 09:53:48.539256 | controller | skipping: Conditional result was False
2026-03-14 09:53:48.568498 | 
2026-03-14 09:53:48.568649 | TASK [env_op_images : Get operator images and pods]
2026-03-14 09:53:48.595092 | controller | skipping: Conditional result was False
2026-03-14 09:53:48.623321 | 
2026-03-14 09:53:48.623425 | LOOP [env_op_images : Add operator images to the dictionary]
2026-03-14 09:53:48.677054 | 
2026-03-14 09:53:48.677269 | TASK [env_op_images : Write images to file]
2026-03-14 09:53:48.714537 | controller | skipping: Conditional result was False
2026-03-14 09:53:48.818791 | 
2026-03-14 09:53:48.818923 | TASK [cifmw_setup : Create a versioned log folder]
2026-03-14 09:53:49.129918 | controller | changed
2026-03-14 09:53:49.158600 | 
2026-03-14 09:53:49.158752 | TASK [cifmw_setup : Return a list of log files in home directory]
2026-03-14 09:53:49.462522 | controller | ok: All paths examined
2026-03-14 09:53:49.490318 | 
2026-03-14 09:53:49.490454 | TASK [cifmw_setup : Ensure ansible facts cache exists]
2026-03-14 09:53:49.803655 | controller | ok
2026-03-14 09:53:49.845358 | 
2026-03-14 09:53:49.845740 | LOOP [cifmw_setup : Copy logs to proper location]
2026-03-14 09:53:50.313226 | controller | changed:
2026-03-14 09:53:50.313515 | controller | {
2026-03-14 09:53:50.313589 | controller |   "atime": 1773481827.3542242,
2026-03-14 09:53:50.313647 | controller |   "ctime": 1773481467.527609,
2026-03-14 09:53:50.313695 | controller |   "dev": 64513,
2026-03-14 09:53:50.313730 | controller |   "gid": 1000,
2026-03-14 09:53:50.313764 | controller |   "gr_name": "zuul",
2026-03-14 09:53:50.313799 | controller |   "inode": 1247249,
2026-03-14 09:53:50.313832 | controller |   "isblk": false,
2026-03-14 09:53:50.313865 | controller |   "ischr": false,
2026-03-14 09:53:50.313898 | controller |   "isdir": false,
2026-03-14 09:53:50.313931 | controller |   "isfifo": false,
2026-03-14 09:53:50.313963 | controller |   "isgid": false,
2026-03-14 09:53:50.313995 | controller |   "islnk": false,
2026-03-14 09:53:50.314071 | controller |   "isreg": true,
2026-03-14 09:53:50.314110 | controller |   "issock": false,
2026-03-14 09:53:50.314145 | controller |   "isuid": false,
2026-03-14 09:53:50.314178 | controller |   "mode": "0644",
2026-03-14 09:53:50.314209 | controller |   "mtime": 1773481467.527609,
2026-03-14 09:53:50.314242 | controller |   "nlink": 1,
2026-03-14 09:53:50.314274 | controller |   "path": "/home/zuul/ansible.log",
2026-03-14 09:53:50.314306 | controller |   "pw_name": "zuul",
2026-03-14 09:53:50.314340 | controller |   "rgrp": true,
2026-03-14 09:53:50.314378 | controller |   "roth": true,
2026-03-14 09:53:50.314415 | controller |   "rusr": true,
2026-03-14 09:53:50.314449 | controller |   "size": 6103908,
2026-03-14 09:53:50.314488 | controller |   "uid": 1000,
2026-03-14 09:53:50.314538 | controller |   "wgrp": false,
2026-03-14 09:53:50.314590 | controller |   "woth": false,
2026-03-14 09:53:50.314645 | controller |   "wusr": true,
2026-03-14 09:53:50.314702 | controller |   "xgrp": false,
2026-03-14 09:53:50.314760 | controller |   "xoth": false,
2026-03-14 09:53:50.314818 | controller |   "xusr": false
2026-03-14 09:53:50.314862 | controller | }
2026-03-14 09:53:50.314916 | 
2026-03-14 09:53:50.364159 | 
2026-03-14 09:53:50.364337 | LOOP [cifmw_setup : Remove original log from home directory]
2026-03-14 09:53:50.627357 | controller | changed:
2026-03-14 09:53:50.627795 | controller | {
2026-03-14 09:53:50.627861 | controller |   "atime": 1773481827.3542242,
2026-03-14 09:53:50.627901 | controller |   "ctime": 1773481467.527609,
2026-03-14 09:53:50.627937 | controller |   "dev": 64513,
2026-03-14 09:53:50.627971 | controller |   "gid": 1000,
2026-03-14 09:53:50.628003 | controller |   "gr_name": "zuul",
2026-03-14 09:53:50.628084 | controller |   "inode": 1247249,
2026-03-14 09:53:50.628119 | controller |   "isblk": false,
2026-03-14 09:53:50.628164 | controller |   "ischr": false,
2026-03-14 09:53:50.628197 | controller |   "isdir": false,
2026-03-14 09:53:50.628229 | controller |   "isfifo": false,
2026-03-14 09:53:50.628262 | controller |   "isgid": false,
2026-03-14 09:53:50.628293 | controller |   "islnk": false,
2026-03-14 09:53:50.628318 | controller |   "isreg": true,
2026-03-14 09:53:50.628338 | controller |   "issock": false,
2026-03-14 09:53:50.628358 | controller |   "isuid": false,
2026-03-14 09:53:50.628378 | controller |   "mode": "0644",
2026-03-14 09:53:50.628398 | controller |   "mtime": 1773481467.527609,
2026-03-14 09:53:50.628418 | controller |   "nlink": 1,
2026-03-14 09:53:50.628439 | controller |   "path": "/home/zuul/ansible.log",
2026-03-14 09:53:50.628459 | controller |   "pw_name": "zuul",
2026-03-14 09:53:50.628479 | controller |   "rgrp": true,
2026-03-14 09:53:50.628499 | controller |   "roth": true,
2026-03-14 09:53:50.628518 | controller |   "rusr": true,
2026-03-14 09:53:50.628537 | controller |   "size": 6103908,
2026-03-14 09:53:50.628558 | controller |   "uid": 1000,
2026-03-14 09:53:50.628578 | controller |   "wgrp": false,
2026-03-14 09:53:50.628599 | controller |   "woth": false,
2026-03-14 09:53:50.628618 | controller |   "wusr": true,
2026-03-14 09:53:50.628638 | controller |   "xgrp": false,
2026-03-14 09:53:50.628658 | controller |   "xoth": false,
2026-03-14 09:53:50.628677 | controller |   "xusr": false
2026-03-14 09:53:50.628697 | controller | }
2026-03-14 09:53:50.679223 | 
2026-03-14 09:53:50.679448 | TASK [cifmw_setup : Copy facts to dated directory]
2026-03-14 09:53:51.103735 | controller | changed
2026-03-14 09:53:51.130141 | 
2026-03-14 09:53:51.130249 | TASK [run_hook : Assert parameters are valid]
2026-03-14 09:53:51.434205 | controller | ok: All assertions passed
2026-03-14 09:53:51.451378 | 
2026-03-14 09:53:51.451462 | TASK [run_hook : Assert single hooks are all mappings]
2026-03-14 09:53:51.609346 | controller | ok: All assertions passed
2026-03-14 09:53:51.633441 | 
2026-03-14 09:53:51.633524 | LOOP [run_hook : Loop on hooks for post_logs]
2026-03-14 09:53:51.870309 | 
2026-03-14 09:53:51.870512 | PLAY [Run ci/playbooks/e2e-collect-logs.yml on CRC host]
2026-03-14 09:53:51.891029 | 
2026-03-14 09:53:51.891099 | TASK [Get kubelet journalctl logs]
2026-03-14 09:53:52.934281 | crc | Output suppressed because no_log was given
2026-03-14 09:53:52.950190 | 
2026-03-14 09:53:52.950342 | TASK [Compress logs bigger than 2MB]
2026-03-14 09:53:54.071523 | crc | changed
2026-03-14 09:53:54.087247 | 
2026-03-14 09:53:54.087384 | TASK [Copy files from workspace on node]
2026-03-14 09:53:54.131076 | crc | ok
2026-03-14 09:53:54.175835 | 
2026-03-14 09:53:54.175957 | TASK [fetch-output : Set log path for multiple nodes]
2026-03-14 09:53:54.251949 | crc | ok
2026-03-14 09:53:54.261783 | 
2026-03-14 09:53:54.261878 | TASK [fetch-output : Set log path for single node]
2026-03-14 09:53:54.287729 | crc | skipping: Conditional result was False
2026-03-14 09:53:54.297533 | 
2026-03-14 09:53:54.297605 | LOOP [fetch-output : Ensure local output dirs]
2026-03-14 09:53:54.550214 | crc -> localhost | changed: "/var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/logs/crc"
2026-03-14 09:53:54.550556 | 
2026-03-14 09:53:54.808674 | crc -> localhost | changed: "/var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/artifacts"
2026-03-14 09:53:55.041561 | crc -> localhost | changed: "/var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/docs"
2026-03-14 09:53:55.071412 | 
2026-03-14 09:53:55.071610 | LOOP [fetch-output : Collect logs, artifacts and docs]
2026-03-14 09:53:55.757948 | crc | changed:
2026-03-14 09:53:55.758587 | crc | .d..t...... ./
2026-03-14 09:53:55.758693 | crc | >f+++++++++ kubelet.log.gz
2026-03-14 09:53:56.264540 | crc | changed: .d..t...... ./
2026-03-14 09:53:56.793082 | crc | changed: .d..t...... ./
2026-03-14 09:53:56.856292 | 
2026-03-14 09:53:56.856414 | PLAY RECAP
2026-03-14 09:53:56.856474 | controller | ok: 75 changed: 34 unreachable: 0 failed: 0 skipped: 39 rescued: 0 ignored: 2
2026-03-14 09:53:56.856524 | crc | ok: 5 changed: 4 unreachable: 0 failed: 0 skipped: 1 rescued: 0 ignored: 0
2026-03-14 09:53:56.856550 | 
2026-03-14 09:53:57.058642 | POST-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/e2e-collect-logs.yml@main]
2026-03-14 09:53:57.063105 | POST-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml@main]
2026-03-14 09:53:57.645638 | 
2026-03-14 09:53:57.645794 | PLAY [Run ci/playbooks/collect-logs.yml]
2026-03-14 09:53:57.758141 | 
2026-03-14 09:53:57.758291 | TASK [Filter out host if needed]
2026-03-14 09:53:57.787326 | controller | skipping: end_host conditional evaluated to False, continuing execution for controller
2026-03-14 09:53:57.794254 | 
2026-03-14 09:53:57.794326 | TASK [Filter out host if needed]
2026-03-14 09:53:57.825822 | 
2026-03-14 09:53:57.825916 | TASK [Ensure file is present]
2026-03-14 09:53:58.344394 | controller | ok
2026-03-14 09:53:58.379612 | 
2026-03-14 09:53:58.379775 | TASK [Manage molecule report file]
2026-03-14 09:53:59.097655 | controller | skipping: Conditional result was False
2026-03-14 09:53:59.150055 | 
2026-03-14 09:53:59.150182 | TASK [Check if we get ci-framework-data basedir]
2026-03-14 09:53:59.402648 | controller | ok
2026-03-14 09:53:59.469054 | 
2026-03-14 09:53:59.469199 | TASK [Create ci-framework-data log directory for zuul]
2026-03-14 09:53:59.855852 | controller | changed
2026-03-14 09:53:59.895389 | 
2026-03-14 09:53:59.895499 | TASK [Copy ci-framework interesting files]
2026-03-14 09:54:00.547328 | controller | cp: cannot stat '/home/zuul/ci-framework-data/tests': No such file or directory
2026-03-14 09:54:10.607193 | controller | changed
2026-03-14 09:54:10.657961 | 
2026-03-14 09:54:10.658125 | TASK [Get SELinux listing]
2026-03-14 09:54:21.377539 | controller | changed
2026-03-14 09:54:21.437147 | 
2026-03-14 09:54:21.437337 | TASK [Generate log index]
2026-03-14 09:54:22.388836 | controller | changed
2026-03-14 09:54:22.417737 | 
2026-03-14 09:54:22.417867 | TASK [Get some env related data]
2026-03-14 09:54:23.268045 | controller | /home/zuul/.local/bin/ansible
2026-03-14 09:54:34.154122 | controller | changed
2026-03-14 09:54:34.206339 | 
2026-03-14 09:54:34.206482 | TASK [Generate list of logs to collect in home directory]
2026-03-14 09:54:34.602329 | controller | ok: All paths examined
2026-03-14 09:54:34.635685 | 
2026-03-14 09:54:34.635833 | LOOP [Copy logs from home directory]
2026-03-14 09:54:34.685478 | 
2026-03-14 09:54:34.685625 | TASK [Copy crio stats log file]
2026-03-14 09:54:34.710003 | controller | skipping: Conditional result was False
2026-03-14 09:54:34.734895 | 
2026-03-14 09:54:34.734999 | TASK [Get SELinux related data]
2026-03-14 09:54:35.148856 | controller | <no matches>
2026-03-14 09:54:45.448590 | controller | ERROR
2026-03-14 09:54:45.448971 | controller | {
2026-03-14 09:54:45.449077 | controller |   "delta": "0:00:00.008178",
2026-03-14 09:54:45.449125 | controller |   "end": "2026-03-14 09:54:35.149664",
2026-03-14 09:54:45.449166 | controller |   "msg": "non-zero return code",
2026-03-14 09:54:45.449204 | controller |   "rc": 1,
2026-03-14 09:54:45.449241 | controller |   "start": "2026-03-14 09:54:35.141486"
2026-03-14 09:54:45.449277 | controller | }
2026-03-14 09:54:45.449325 | controller | ERROR: Ignoring Errors
2026-03-14 09:54:45.504480 | 
2026-03-14 09:54:45.504597 | TASK [Create system configuration directory]
2026-03-14 09:54:45.806951 | controller | changed
2026-03-14 09:54:45.830882 | 
2026-03-14 09:54:45.830982 | TASK [Get some of the system configurations]
2026-03-14 09:54:56.526753 | controller | changed
2026-03-14 09:54:56.579552 | 
2026-03-14 09:54:56.579718 | TASK [Copy generated documentation if available]
2026-03-14 09:54:56.607660 | controller | skipping: Conditional result was False
2026-03-14 09:54:56.645633 | 
2026-03-14 09:54:56.645889 | TASK [Copy generated AsciiDoc documentation if available]
2026-03-14 09:54:56.672971 | controller | skipping: Conditional result was False
2026-03-14 09:54:56.708967 | 
2026-03-14 09:54:56.709107 | TASK [Compress logs bigger than 2MB]
2026-03-14 09:55:25.439600 | controller | changed
2026-03-14 09:55:25.485477 | 
2026-03-14 09:55:25.485583 | TASK [Copy files from workspace on node]
2026-03-14 09:55:25.577360 | controller | ok
2026-03-14 09:55:25.670675 | 
2026-03-14 09:55:25.670852 | TASK [fetch-output : Set log path for multiple nodes]
2026-03-14 09:55:25.755675 | controller | ok
2026-03-14 09:55:25.826586 | 
2026-03-14 09:55:25.826754 | TASK [fetch-output : Set log path for single node]
2026-03-14 09:55:25.854032 | controller | skipping: Conditional result was False
2026-03-14 09:55:25.880210 | 
2026-03-14 09:55:25.880349 | LOOP [fetch-output : Ensure local output dirs]
2026-03-14 09:55:26.150797 | controller -> localhost | changed: "/var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/logs/controller"
2026-03-14 09:55:26.151107 | 
2026-03-14 09:55:26.342623 | controller -> localhost | ok: "/var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/artifacts"
2026-03-14 09:55:26.531147 | controller -> localhost | ok: "/var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/docs"
2026-03-14 09:55:26.567377 | 
2026-03-14 09:55:26.567533 | LOOP [fetch-output : Collect logs, artifacts and docs]
2026-03-14 09:55:30.418513 | controller | changed:
2026-03-14 09:55:30.418907 | controller | .d..t...... ./
2026-03-14 09:55:30.418946 | controller | >f+++++++++ README.html
2026-03-14 09:55:30.418970 | controller | >f+++++++++ dmesg.log
2026-03-14 09:55:30.418993 | controller | >f+++++++++ installed-pkgs.log
2026-03-14 09:55:30.419014 | controller | >f+++++++++ python.log
2026-03-14 09:55:30.419073 | controller | >f+++++++++ registries.conf
2026-03-14 09:55:30.419097 | controller | >f+++++++++ selinux-denials.log
2026-03-14 09:55:30.419117 | controller | >f+++++++++ selinux-listing.log
2026-03-14 09:55:30.419139 | controller | cd+++++++++ ci-framework-data/
2026-03-14 09:55:30.419176 | controller | cd+++++++++ ci-framework-data/artifacts/
2026-03-14 09:55:30.419197 | controller | >f+++++++++ ci-framework-data/artifacts/ansible-facts.yml
2026-03-14 09:55:30.419216 | controller | >f+++++++++ ci-framework-data/artifacts/ansible-vars.yml
2026-03-14 09:55:30.419235 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_check_for_oc.sh
2026-03-14 09:55:30.419457 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_copy_logs_from_crc.sh
2026-03-14 09:55:30.419488 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_prepare_root_ssh.sh
2026-03-14 09:55:30.419509 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_run_download.sh
2026-03-14 09:55:30.419529 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_run_openstack_must_gather.sh
2026-03-14 09:55:30.419550 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_001_run_hook_without_retry.sh
2026-03-14 09:55:30.419573 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_002_fetch_openshift.sh
2026-03-14 09:55:30.419590 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_003_run_openstack.sh
2026-03-14 09:55:30.419609 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_004_run_hook_without_retry_deploy.sh
2026-03-14 09:55:30.419628 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_005_stop_watcher.sh
2026-03-14 09:55:30.419651 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_006_install_watcher_operator_in.sh
2026-03-14 09:55:30.419669 | controller | >f+++++++++ ci-framework-data/artifacts/hosts
2026-03-14 09:55:30.419688 | controller | >f+++++++++ ci-framework-data/artifacts/install_yamls.sh
2026-03-14 09:55:30.419706 | controller | >f+++++++++ ci-framework-data/artifacts/installed-packages.yml
2026-03-14 09:55:30.419725 | controller | >f+++++++++ ci-framework-data/artifacts/ip-network.txt
2026-03-14 09:55:30.419742 | controller | >f+++++++++ ci-framework-data/artifacts/resolv.conf
2026-03-14 09:55:30.419761 | controller | >f+++++++++ ci-framework-data/artifacts/zuul_inventory.yml
2026-03-14 09:55:30.419912 | controller | cd+++++++++ ci-framework-data/artifacts/NetworkManager/
2026-03-14 09:55:30.419936 | controller | >f+++++++++ ci-framework-data/artifacts/NetworkManager/ci-private-network.nmconnection
2026-03-14 09:55:30.419955 | controller | >f+++++++++ ci-framework-data/artifacts/NetworkManager/ens3.nmconnection
2026-03-14 09:55:30.419981 | controller | cd+++++++++ ci-framework-data/artifacts/ansible_facts.2026-03-14_09-53/
2026-03-14 09:55:30.419999 | controller | cd+++++++++ ci-framework-data/artifacts/ansible_facts.2026-03-14_09-53/ansible_facts_cache/
2026-03-14 09:55:30.420047 | controller | >f+++++++++ ci-framework-data/artifacts/ansible_facts.2026-03-14_09-53/ansible_facts_cache/controller
2026-03-14 09:55:30.420069 | controller | >f+++++++++ ci-framework-data/artifacts/ansible_facts.2026-03-14_09-53/ansible_facts_cache/localhost
2026-03-14 09:55:30.420088 | controller | cd+++++++++ ci-framework-data/artifacts/ci-env/
2026-03-14 09:55:30.420107 | controller | >f+++++++++ ci-framework-data/artifacts/ci-env/networking-info.yml
2026-03-14 09:55:30.420126 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/
2026-03-14 09:55:30.420149 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/
2026-03-14 09:55:30.420171 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/namespace.yaml
2026-03-14 09:55:30.420190 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/cert-manager/
2026-03-14 09:55:30.420216 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/cert-manager/op/
2026-03-14 09:55:30.420235 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/cert-manager/op/operatorgroup.yaml
2026-03-14 09:55:30.420255 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/cert-manager/op/subscription.yaml
2026-03-14 09:55:30.420274 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/metallb-system/
2026-03-14 09:55:30.420294 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/namespace.yaml
2026-03-14 09:55:30.420313 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/
2026-03-14 09:55:30.420334 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/
2026-03-14 09:55:30.420356 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/bgpadvertisement.yaml
2026-03-14 09:55:30.420375 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/bgpextras.yaml
2026-03-14 09:55:30.420394 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/bgppeers.yaml
2026-03-14 09:55:30.420413 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/deploy_operator.yaml
2026-03-14 09:55:30.420432 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/ipaddresspools.yaml
2026-03-14 09:55:30.420451 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/l2advertisement.yaml
2026-03-14 09:55:30.420469 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/
2026-03-14 09:55:30.420488 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/namespace.yaml
2026-03-14 09:55:30.420507 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/nmstate/
2026-03-14 09:55:30.420959 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/nmstate/cr/
2026-03-14 09:55:30.420996 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/nmstate/cr/deploy_operator.yaml
2026-03-14 09:55:30.421038 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/
2026-03-14 09:55:30.421061 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/namespace.yaml
2026-03-14 09:55:30.421086 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/metallb/
2026-03-14 09:55:30.421109 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/metallb/op/
2026-03-14 09:55:30.421128 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/metallb/op/operatorgroup.yaml
2026-03-14 09:55:30.421146 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/metallb/op/subscription.yaml
2026-03-14 09:55:30.421163 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/nmstate/
2026-03-14 09:55:30.421182 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/nmstate/op/
2026-03-14 09:55:30.421212 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/nmstate/op/operatorgroup.yaml
2026-03-14 09:55:30.421231 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/nmstate/op/subscription.yaml
2026-03-14 09:55:30.421250 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/
2026-03-14 09:55:30.421268 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/op/
2026-03-14 09:55:30.421286 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/op/catalogsource.yaml
2026-03-14 09:55:30.421304 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/op/operatorgroup.yaml
2026-03-14 09:55:30.421500 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/op/subscription.yaml
2026-03-14 09:55:30.421525 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/
2026-03-14 09:55:30.421544 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/namespace.yaml
2026-03-14 09:55:30.421563 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/cr/
2026-03-14 09:55:30.421583 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/
2026-03-14 09:55:30.421601 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/
2026-03-14 09:55:30.421619 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/ctlplane.yaml
2026-03-14 09:55:30.421638 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/datacentre.yaml
2026-03-14 09:55:30.421656 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/designate.yaml
2026-03-14 09:55:30.421675 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/designateext.yaml
2026-03-14 09:55:30.421697 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/internalapi.yaml
2026-03-14 09:55:30.421718 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/octavia.yaml
2026-03-14 09:55:30.421737 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/storage.yaml
2026-03-14 09:55:30.421755 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/storagemgmt.yaml
2026-03-14 09:55:30.421773 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/tenant.yaml
2026-03-14 09:55:30.421792 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/nncp/
2026-03-14 09:55:30.421810 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/nncp/cr/
2026-03-14 09:55:30.421829 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/nncp/cr/crc_nncp.yaml
2026-03-14 09:55:30.421847 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/openstack/
2026-03-14 09:55:30.421865 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/openstack/cr/
2026-03-14 09:55:30.421889 | controller | cd+++++++++ ci-framework-data/artifacts/parameters/
2026-03-14 09:55:30.421908 | controller | >f+++++++++ ci-framework-data/artifacts/parameters/custom-params.yml
2026-03-14 09:55:30.421927 | controller | >f+++++++++ ci-framework-data/artifacts/parameters/install-yamls-params.yml
2026-03-14 09:55:30.421945 | controller | >f+++++++++ ci-framework-data/artifacts/parameters/openshift-login-params.yml
2026-03-14 09:55:30.421963 | controller | >f+++++++++ ci-framework-data/artifacts/parameters/zuul-params.yml
2026-03-14 09:55:30.421981 | controller | cd+++++++++ ci-framework-data/artifacts/repositories/
2026-03-14 09:55:30.422000 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/delorean-antelope-testing.repo
2026-03-14 09:55:30.422059 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/delorean.repo
2026-03-14 09:55:30.422084 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/delorean.repo.md5
2026-03-14 09:55:30.422103 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/repo-setup-centos-appstream.repo
2026-03-14 09:55:30.422122 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/repo-setup-centos-baseos.repo
2026-03-14 09:55:30.422143 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/repo-setup-centos-highavailability.repo
2026-03-14 09:55:30.422161 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/repo-setup-centos-powertools.repo
2026-03-14 09:55:30.422183 | controller | cd+++++++++ ci-framework-data/artifacts/roles/
2026-03-14 09:55:30.422202 | controller | cd+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/
2026-03-14 09:55:30.422222 | controller | cd+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/
2026-03-14 09:55:30.422241 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_all.yml
2026-03-14 09:55:30.422259 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_attach_default_interface.yml
2026-03-14 09:55:30.422277 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_attach_default_interface_cleanup.yml
2026-03-14 09:55:30.422296 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican.yml
2026-03-14 09:55:30.422424 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_cleanup.yml
2026-03-14 09:55:30.422448 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_deploy.yml
2026-03-14 09:55:30.422468 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_deploy_cleanup.yml
2026-03-14 09:55:30.422488 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_deploy_prep.yml
2026-03-14 09:55:30.422507 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_deploy_validate.yml
2026-03-14 09:55:30.422526 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_kuttl.yml
2026-03-14 09:55:30.422545 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_kuttl_run.yml
2026-03-14 09:55:30.422564 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_prep.yml
2026-03-14 09:55:30.422583 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_baremetal.yml
2026-03-14 09:55:30.422602 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_baremetal_cleanup.yml
2026-03-14 09:55:30.422621 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_baremetal_prep.yml
2026-03-14 09:55:30.422640 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas.yml
2026-03-14 09:55:30.422664 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_baremetal_net_nad.yml
2026-03-14 09:55:30.422684 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_baremetal_net_nad_cleanup.yml
2026-03-14 09:55:30.422702 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_cleanup.yml
2026-03-14 09:55:30.422720 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_crc_attach_network.yml
2026-03-14 09:55:30.422739 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_crc_attach_network_cleanup.yml
2026-03-14 09:55:30.422757 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_crc_baremetal_bridge.yml
2026-03-14 09:55:30.422775 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_crc_baremetal_bridge_cleanup.yml
2026-03-14 09:55:30.422794 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_generate_nodes_yaml.yml
2026-03-14 09:55:30.422812 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_metallb.yml
2026-03-14 09:55:30.422831 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_metallb_cleanup.yml
2026-03-14 09:55:30.422849 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_network.yml
2026-03-14 09:55:30.422868 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_network_cleanup.yml
2026-03-14 09:55:30.422886 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_route_crc_and_crc_bmaas_networks.yml
2026-03-14 09:55:30.422904 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_route_crc_and_crc_bmaas_networks_cleanup.yml
2026-03-14 09:55:30.422923 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_sushy_emulator.yml
2026-03-14 09:55:30.422941 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_sushy_emulator_cleanup.yml
2026-03-14 09:55:30.422959 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_sushy_emulator_wait.yml
2026-03-14 09:55:30.422978 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_virtual_bms.yml
2026-03-14 09:55:30.422996 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_virtual_bms_cleanup.yml
2026-03-14 09:55:30.423043 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ceph.yml
2026-03-14 09:55:30.423071 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ceph_cleanup.yml
2026-03-14 09:55:30.423090 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ceph_help.yml
2026-03-14 09:55:30.423109 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_certmanager.yml
2026-03-14 09:55:30.423126 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_certmanager_cleanup.yml
2026-03-14 09:55:30.423156 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cifmw_cleanup.yml
2026-03-14 09:55:30.423176 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cifmw_prepare.yml
2026-03-14 09:55:30.423195 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder.yml
2026-03-14 09:55:30.423213 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_cleanup.yml
2026-03-14 09:55:30.423236 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_deploy.yml
2026-03-14 09:55:30.423260 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_deploy_cleanup.yml
2026-03-14 09:55:30.423279 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_deploy_prep.yml
2026-03-14 09:55:30.423298 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_kuttl.yml
2026-03-14 09:55:30.423316 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_kuttl_run.yml
2026-03-14 09:55:30.423334 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_prep.yml
2026-03-14 09:55:30.423352 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cleanup.yml
2026-03-14 09:55:30.423371 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc.yml
2026-03-14 09:55:30.423389 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_attach_default_interface.yml
2026-03-14 09:55:30.423408 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_attach_default_interface_cleanup.yml
2026-03-14 09:55:30.423426 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_bmo_cleanup.yml
2026-03-14 09:55:30.423444 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_bmo_setup.yml
2026-03-14 09:55:30.423462 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_cleanup.yml
2026-03-14 09:55:30.423480 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_scrub.yml
2026-03-14 09:55:30.423500 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage.yml
2026-03-14 09:55:30.423524 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage_cleanup.yml
2026-03-14 09:55:30.423543 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage_cleanup_with_retries.yml
2026-03-14 09:55:30.423562 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage_release.yml
2026-03-14 09:55:30.423582 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage_with_retries.yml
2026-03-14 09:55:30.423600 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_deploy_cleanup.yml
2026-03-14 09:55:30.423619 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate.yml
2026-03-14 09:55:30.423637 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_cleanup.yml
2026-03-14 09:55:30.423655 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_deploy.yml
2026-03-14 09:55:30.423674 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_deploy_cleanup.yml
2026-03-14 09:55:30.423694 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_deploy_prep.yml
2026-03-14 09:55:30.423713 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_kuttl.yml
2026-03-14 09:55:30.423731 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_kuttl_run.yml
2026-03-14 09:55:30.423750 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_prep.yml
2026-03-14 09:55:30.423768 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_dns_deploy.yml
2026-03-14 09:55:30.423786 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_dns_deploy_cleanup.yml
2026-03-14 09:55:30.423810 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_dns_deploy_prep.yml
2026-03-14 09:55:30.423828 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_download_tools.yml
2026-03-14 09:55:30.423846 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_ansible_runner.yml
2026-03-14 09:55:30.423869 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_baremetal_compute.yml
2026-03-14 09:55:30.423887 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_compute.yml
2026-03-14 09:55:30.423907 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_compute_bootc.yml
2026-03-14 09:55:30.423924 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_compute_cleanup.yml
2026-03-14 09:55:30.423944 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_compute_repos.yml
2026-03-14 09:55:30.423962 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_computes_bgp.yml
2026-03-14 09:55:30.423982 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy.yml
2026-03-14 09:55:30.424000 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_baremetal.yml
2026-03-14 09:55:30.424054 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_baremetal_prep.yml
2026-03-14 09:55:30.424089 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_cleanup.yml
2026-03-14 09:55:30.424110 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_generate_keys.yml
2026-03-14 09:55:30.424136 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_instance.yml
2026-03-14 09:55:30.424156 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_networker.yml
2026-03-14 09:55:30.424175 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_networker_cleanup.yml
2026-03-14 09:55:30.424193 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_networker_prep.yml
2026-03-14 09:55:30.424211 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_prep.yml
2026-03-14 09:55:30.424229 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_networker.yml
2026-03-14 09:55:30.424248 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_networker_cleanup.yml
2026-03-14 09:55:30.424266 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_nova_discover_hosts.yml
2026-03-14 09:55:30.424285 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_patch_ansible_runner_image.yml
2026-03-14 09:55:30.424303 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_register_dns.yml
2026-03-14 09:55:30.424321 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_wait_deploy.yml
2026-03-14 09:55:30.424339 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_wait_deploy_baremetal.yml
2026-03-14 09:55:30.424357 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance.yml
2026-03-14 09:55:30.424375 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_cleanup.yml
2026-03-14 09:55:30.424401 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_deploy.yml
2026-03-14 09:55:30.424420 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_deploy_cleanup.yml
2026-03-14 09:55:30.424438 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_deploy_prep.yml
2026-03-14 09:55:30.424456 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_kuttl.yml
2026-03-14 09:55:30.424474 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_kuttl_run.yml
2026-03-14 09:55:30.424493 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_prep.yml
2026-03-14 09:55:30.424510 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat.yml
2026-03-14 09:55:30.424529 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_cleanup.yml
2026-03-14 09:55:30.424546 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_deploy.yml
2026-03-14 09:55:30.424564 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_deploy_cleanup.yml
2026-03-14 09:55:30.424582 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_deploy_prep.yml
2026-03-14 09:55:30.424600 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_kuttl.yml
2026-03-14 09:55:30.424618 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_kuttl_crc.yml
2026-03-14 09:55:30.424636 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_kuttl_run.yml
2026-03-14 09:55:30.424658 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_prep.yml
2026-03-14 09:55:30.424676 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_help.yml
2026-03-14 09:55:30.424694 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon.yml
2026-03-14 09:55:30.424712 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_cleanup.yml
2026-03-14 09:55:30.424732 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_deploy.yml
2026-03-14 09:55:30.424751 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_deploy_cleanup.yml
2026-03-14 09:55:30.424770 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_deploy_prep.yml
2026-03-14 09:55:30.424787 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_kuttl.yml
2026-03-14 09:55:30.424805 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_kuttl_run.yml
2026-03-14 09:55:30.424823 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_prep.yml
2026-03-14 09:55:30.424840 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra.yml
2026-03-14 09:55:30.424858 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra_cleanup.yml
2026-03-14 09:55:30.424875 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra_kuttl.yml
2026-03-14 09:55:30.424893 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra_kuttl_run.yml
2026-03-14 09:55:30.424912 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra_prep.yml
2026-03-14 09:55:30.424933 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_input.yml
2026-03-14 09:55:30.424960 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_input_cleanup.yml
2026-03-14 09:55:30.424984 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab.yml
2026-03-14 09:55:30.425002 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_cleanup.yml
2026-03-14 09:55:30.425043 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_nat64_router.yml
2026-03-14 09:55:30.425064 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_nat64_router_cleanup.yml
2026-03-14 09:55:30.425082 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_network.yml
2026-03-14 09:55:30.425101 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_network_cleanup.yml
2026-03-14 09:55:30.425127 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_sno.yml
2026-03-14 09:55:30.425148 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_sno_cleanup.yml
2026-03-14 09:55:30.425167 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic.yml
2026-03-14 09:55:30.425185 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_cleanup.yml
2026-03-14 09:55:30.425202 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_deploy.yml
2026-03-14 09:55:30.425220 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_deploy_cleanup.yml
2026-03-14 09:55:30.425239 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_deploy_prep.yml
2026-03-14 09:55:30.425256 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_kuttl.yml
2026-03-14 09:55:30.425378 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_kuttl_crc.yml
2026-03-14 09:55:30.425402 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_kuttl_run.yml
2026-03-14 09:55:30.425422 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_prep.yml
2026-03-14 09:55:30.425442 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone.yml
2026-03-14 09:55:30.425460 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_cleanup.yml
2026-03-14 09:55:30.425479 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_deploy.yml
2026-03-14 09:55:30.425497 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_deploy_cleanup.yml
2026-03-14 09:55:30.425517 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_deploy_prep.yml
2026-03-14 09:55:30.425536 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_kuttl.yml
2026-03-14 09:55:30.425556 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_kuttl_run.yml
2026-03-14 09:55:30.425575 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_prep.yml
2026-03-14 09:55:30.425594 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_kuttl_common_cleanup.yml
2026-03-14 09:55:30.425685 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_kuttl_common_prep.yml
2026-03-14 09:55:30.425719 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_kuttl_db_cleanup.yml
2026-03-14 09:55:30.425741 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_kuttl_db_prep.yml
2026-03-14 09:55:30.425769 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_loki.yml
2026-03-14 09:55:30.425789 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_loki_cleanup.yml
2026-03-14 09:55:30.425810 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_loki_deploy.yml
2026-03-14 09:55:30.425830 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_loki_deploy_cleanup.yml
2026-03-14 09:55:30.425849 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_lvms.yml
2026-03-14 09:55:30.425870 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila.yml
2026-03-14 09:55:30.425890 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_cleanup.yml
2026-03-14 09:55:30.425911 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_deploy.yml
2026-03-14 09:55:30.425929 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_deploy_cleanup.yml
2026-03-14 09:55:30.425947 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_deploy_prep.yml
2026-03-14 09:55:30.425965 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_kuttl.yml
2026-03-14 09:55:30.425984 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_kuttl_run.yml
2026-03-14 09:55:30.426287 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_prep.yml
2026-03-14 09:55:30.426327 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb.yml
2026-03-14 09:55:30.426348 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_chainsaw.yml
2026-03-14 09:55:30.426365 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_chainsaw_run.yml
2026-03-14 09:55:30.426380 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_cleanup.yml
2026-03-14 09:55:30.426397 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_deploy.yml
2026-03-14 09:55:30.426413 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_deploy_cleanup.yml
2026-03-14 09:55:30.426428 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_deploy_prep.yml
2026-03-14 09:55:30.426444 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_kuttl.yml
2026-03-14 09:55:30.426459 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_kuttl_run.yml
2026-03-14 09:55:30.426474 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_memcached_deploy.yml
2026-03-14 09:55:30.426490 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_memcached_deploy_cleanup.yml
2026-03-14 09:55:30.426505 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_memcached_deploy_prep.yml
2026-03-14 09:55:30.426520 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_metallb.yml
2026-03-14 09:55:30.426535 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_metallb_cleanup.yml
2026-03-14 09:55:30.426551 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_metallb_config.yml
2026-03-14 09:55:30.426659 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_metallb_config_cleanup.yml
2026-03-14 09:55:30.426688 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mirror_registry.yml
2026-03-14 09:55:30.426706 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mirror_registry_ca.yml
2026-03-14 09:55:30.426722 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mirror_registry_cleanup.yml
2026-03-14 09:55:30.426738 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mirror_registry_secure.yml
2026-03-14 09:55:30.426753 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_namespace.yml
2026-03-14 09:55:30.426769 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_namespace_cleanup.yml
2026-03-14 09:55:30.426784 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netattach.yml
2026-03-14 09:55:30.426800 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netattach_cleanup.yml
2026-03-14 09:55:30.426815 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netconfig_deploy.yml
2026-03-14 09:55:30.426830 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netconfig_deploy_cleanup.yml
2026-03-14 09:55:30.426845 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netconfig_deploy_prep.yml
2026-03-14 09:55:30.426860 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netobserv.yml
2026-03-14 09:55:30.426876 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netobserv_cleanup.yml
2026-03-14 09:55:30.426892 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netobserv_deploy.yml
2026-03-14 09:55:30.426907 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netobserv_deploy_cleanup.yml
2026-03-14 09:55:30.427104 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_network_isolation_bridge.yml
2026-03-14 09:55:30.427146 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_network_isolation_bridge_cleanup.yml
2026-03-14 09:55:30.427166 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron.yml
2026-03-14 09:55:30.427183 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_cleanup.yml
2026-03-14 09:55:30.427199 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_deploy.yml
2026-03-14 09:55:30.427215 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_deploy_cleanup.yml
2026-03-14 09:55:30.427230 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_deploy_prep.yml
2026-03-14 09:55:30.427246 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_kuttl.yml
2026-03-14 09:55:30.427261 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_kuttl_run.yml
2026-03-14 09:55:30.427393 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_prep.yml
2026-03-14 09:55:30.427411 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nfs.yml
2026-03-14 09:55:30.427428 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nfs_cleanup.yml
2026-03-14 09:55:30.427444 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nmstate.yml
2026-03-14 09:55:30.427460 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nncp.yml
2026-03-14 09:55:30.427476 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nncp_cleanup.yml
2026-03-14 09:55:30.427496 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova.yml
2026-03-14 09:55:30.427512 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_cleanup.yml
2026-03-14 09:55:30.427528 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_deploy.yml
2026-03-14 09:55:30.427543 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_deploy_cleanup.yml
2026-03-14 09:55:30.427558 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_deploy_prep.yml
2026-03-14 09:55:30.427573 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_prep.yml
2026-03-14 09:55:30.427588 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia.yml
2026-03-14 09:55:30.427604 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_cleanup.yml
2026-03-14 09:55:30.427619 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_deploy.yml
2026-03-14 09:55:30.427634 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_deploy_cleanup.yml
2026-03-14 09:55:30.427649 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_deploy_prep.yml
2026-03-14 09:55:30.427665 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_kuttl.yml
2026-03-14 09:55:30.427680 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_kuttl_run.yml
2026-03-14 09:55:30.427696 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_prep.yml
2026-03-14 09:55:30.427712 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack.yml
2026-03-14 09:55:30.427727 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_cleanup.yml
2026-03-14 09:55:30.427742 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_crds.yml
2026-03-14 09:55:30.427757 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_crds_cleanup.yml
2026-03-14 09:55:30.427771 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_deploy.yml
2026-03-14 09:55:30.427786 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_deploy_cleanup.yml
2026-03-14 09:55:30.427808 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_deploy_prep.yml
2026-03-14 09:55:30.427829 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_init.yml
2026-03-14 09:55:30.427848 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_kuttl.yml
2026-03-14 09:55:30.427864 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_kuttl_cleanup.yml
2026-03-14 09:55:30.427879 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_kuttl_prep.yml
2026-03-14 09:55:30.428492 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_kuttl_run.yml
2026-03-14 09:55:30.428524 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_patch_version.yml
2026-03-14 09:55:30.428546 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_prep.yml
2026-03-14 09:55:30.428567 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_repo.yml
2026-03-14 09:55:30.428731 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_update_run.yml
2026-03-14 09:55:30.428764 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_wait.yml
2026-03-14 09:55:30.428783 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_wait_deploy.yml
2026-03-14 09:55:30.428799 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_operator_namespace.yml
2026-03-14 09:55:30.428815 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn.yml
2026-03-14 09:55:30.428831 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_cleanup.yml
2026-03-14 09:55:30.428846 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_deploy.yml
2026-03-14 09:55:30.428861 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_deploy_cleanup.yml
2026-03-14 09:55:30.428877 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_deploy_prep.yml
2026-03-14 09:55:30.428891 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_kuttl.yml
2026-03-14 09:55:30.428906 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_kuttl_run.yml
2026-03-14 09:55:30.428920 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_prep.yml
2026-03-14 09:55:30.428935 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement.yml
2026-03-14 09:55:30.428950 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_cleanup.yml
2026-03-14 09:55:30.428964 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_deploy.yml
2026-03-14 09:55:30.428979 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_deploy_cleanup.yml
2026-03-14 09:55:30.428994 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_deploy_prep.yml
2026-03-14 09:55:30.429009 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_kuttl.yml
2026-03-14 09:55:30.429044 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_kuttl_run.yml
2026-03-14 09:55:30.429061 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_prep.yml
2026-03-14 09:55:30.429077 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq.yml
2026-03-14 09:55:30.429338 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_cleanup.yml
2026-03-14 09:55:30.429363 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_deploy.yml
2026-03-14 09:55:30.429381 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_deploy_cleanup.yml
2026-03-14 09:55:30.429397 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_deploy_prep.yml
2026-03-14 09:55:30.429414 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_prep.yml
2026-03-14 09:55:30.429430 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_redis_deploy.yml
2026-03-14 09:55:30.429445 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_redis_deploy_cleanup.yml
2026-03-14 09:55:30.429461 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_redis_deploy_prep.yml
2026-03-14 09:55:30.429481 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook.yml
2026-03-14 09:55:30.429497 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_cleanup.yml
2026-03-14 09:55:30.429512 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_crc_disk.yml
2026-03-14 09:55:30.429527 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_deploy.yml
2026-03-14 09:55:30.429542 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_deploy_prep.yml
2026-03-14 09:55:30.429639 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_prep.yml
2026-03-14 09:55:30.429668 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_set_slower_etcd_profile.yml
2026-03-14 09:55:30.429686 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone.yml
2026-03-14 09:55:30.429702 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_cleanup.yml
2026-03-14 09:55:30.429718 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_deploy.yml
2026-03-14 09:55:30.429734 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_revert.yml
2026-03-14 09:55:30.429750 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_snapshot.yml
2026-03-14 09:55:30.429765 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_sync.yml
2026-03-14 09:55:30.429781 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift.yml
2026-03-14 09:55:30.429797 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_cleanup.yml
2026-03-14 09:55:30.429812 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_deploy.yml
2026-03-14 09:55:30.429828 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_deploy_cleanup.yml
2026-03-14 09:55:30.429843 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_deploy_prep.yml
2026-03-14 09:55:30.429858 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_kuttl.yml
2026-03-14 09:55:30.429999 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_kuttl_run.yml
2026-03-14 09:55:30.430053 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_prep.yml
2026-03-14 09:55:30.430073 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry.yml
2026-03-14 09:55:30.430089 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_cleanup.yml
2026-03-14 09:55:30.430104 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_deploy.yml
2026-03-14 09:55:30.430120 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_deploy_cleanup.yml
2026-03-14 09:55:30.430136 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_deploy_prep.yml
2026-03-14 09:55:30.430151 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_kuttl.yml
2026-03-14 09:55:30.430166 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_kuttl_run.yml
2026-03-14 09:55:30.430181 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_prep.yml
2026-03-14 09:55:30.430196 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_tripleo_deploy.yml
2026-03-14 09:55:30.430382 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_update_services.yml
2026-03-14 09:55:30.430414 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_update_system.yml
2026-03-14 09:55:30.430432 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_validate_marketplace.yml
2026-03-14 09:55:30.430448 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_wait.yml
2026-03-14 09:55:30.430464 | controller | cd+++++++++ ci-framework-data/artifacts/yum_repos/
2026-03-14 09:55:30.430483 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/delorean-antelope-testing.repo
2026-03-14 09:55:30.430500 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/delorean.repo
2026-03-14 09:55:30.430517 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/delorean.repo.md5
2026-03-14 09:55:30.430533 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/repo-setup-centos-appstream.repo
2026-03-14 09:55:30.430549 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/repo-setup-centos-baseos.repo
2026-03-14 09:55:30.430564 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/repo-setup-centos-highavailability.repo
2026-03-14 09:55:30.430580 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/repo-setup-centos-powertools.repo
2026-03-14 09:55:30.430596 | controller | cd+++++++++ ci-framework-data/logs/
2026-03-14 09:55:30.430730 | controller | >f+++++++++ ci-framework-data/logs/ansible.log.gz
2026-03-14 09:55:30.430758 | controller | >f+++++++++ ci-framework-data/logs/ci_script_000_check_for_oc.log
2026-03-14 09:55:30.430775 | controller | >f+++++++++ ci-framework-data/logs/ci_script_000_copy_logs_from_crc.log
2026-03-14 09:55:30.430792 | controller | >f+++++++++ ci-framework-data/logs/ci_script_000_prepare_root_ssh.log
2026-03-14 09:55:30.430808 | controller | >f+++++++++ ci-framework-data/logs/ci_script_000_run_download.log
2026-03-14 09:55:30.430824 | controller | >f+++++++++ ci-framework-data/logs/ci_script_000_run_openstack_must_gather.log
2026-03-14 09:55:30.430840 | controller | >f+++++++++ ci-framework-data/logs/ci_script_001_run_hook_without_retry.log
2026-03-14 09:55:30.430855 | controller | >f+++++++++ ci-framework-data/logs/ci_script_002_fetch_openshift.log
2026-03-14 09:55:30.430870 | controller | >f+++++++++ ci-framework-data/logs/ci_script_003_run_openstack.log
2026-03-14 09:55:30.430885 | controller | >f+++++++++ ci-framework-data/logs/ci_script_004_run_hook_without_retry_deploy.log
2026-03-14 09:55:30.430901 | controller | >f+++++++++ ci-framework-data/logs/ci_script_005_stop_watcher.log
2026-03-14 09:55:30.430916 | controller | >f+++++++++ ci-framework-data/logs/ci_script_006_install_watcher_operator_in.log
2026-03-14 09:55:30.431106 | controller | >f+++++++++ ci-framework-data/logs/post_install_operators_kuttl_from_operator_deploy_watcher_operator.log
2026-03-14 09:55:30.431142 | controller | >f+++++++++ ci-framework-data/logs/pre_infra_download_needed_tools.log
2026-03-14 09:55:30.431160 | controller | cd+++++++++ ci-framework-data/logs/2026-03-14_09-44/
2026-03-14 09:55:30.431176 | controller | >f+++++++++ ci-framework-data/logs/2026-03-14_09-44/ansible.log.gz
2026-03-14 09:55:30.431192 | controller | cd+++++++++ ci-framework-data/logs/crc/
2026-03-14 09:55:30.431207 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/
2026-03-14 09:55:30.431223 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/
2026-03-14 09:55:30.431239 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager-operator_cert-manager-operator-controller-manager-66c8bdd694-2lhwc_cc8d2a29-78e2-4bc0-9efc-5da547e43e0b/
2026-03-14 09:55:30.431255 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager-operator_cert-manager-operator-controller-manager-66c8bdd694-2lhwc_cc8d2a29-78e2-4bc0-9efc-5da547e43e0b/cert-manager-operator/
2026-03-14 09:55:30.431279 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager-operator_cert-manager-operator-controller-manager-66c8bdd694-2lhwc_cc8d2a29-78e2-4bc0-9efc-5da547e43e0b/cert-manager-operator/0.log
2026-03-14 09:55:30.431296 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-545d4d4674-jmg2k_04227b43-3672-4848-84f8-275b2ec997d8/
2026-03-14 09:55:30.432168 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-545d4d4674-jmg2k_04227b43-3672-4848-84f8-275b2ec997d8/cert-manager-controller/
2026-03-14 09:55:30.432207 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-545d4d4674-jmg2k_04227b43-3672-4848-84f8-275b2ec997d8/cert-manager-controller/0.log
2026-03-14 09:55:30.432227 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-cainjector-5545bd876-lmlng_2ab2a4c0-0a05-4ffa-9900-9af5dbe961df/
2026-03-14 09:55:30.432245 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-cainjector-5545bd876-lmlng_2ab2a4c0-0a05-4ffa-9900-9af5dbe961df/cert-manager-cainjector/
2026-03-14 09:55:30.432262 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-cainjector-5545bd876-lmlng_2ab2a4c0-0a05-4ffa-9900-9af5dbe961df/cert-manager-cainjector/0.log
2026-03-14 09:55:30.432278 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-webhook-6888856db4-hfb75_470dcdf7-6d52-4164-bcfe-c8922d387147/
2026-03-14 09:55:30.432569 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-webhook-6888856db4-hfb75_470dcdf7-6d52-4164-bcfe-c8922d387147/cert-manager-webhook/
2026-03-14 09:55:30.432601 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-webhook-6888856db4-hfb75_470dcdf7-6d52-4164-bcfe-c8922d387147/cert-manager-webhook/0.log
2026-03-14 09:55:30.432620 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-jqf62_60855808-2138-47e7-9539-df7c86ebe635/
2026-03-14 09:55:30.432822 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-jqf62_60855808-2138-47e7-9539-df7c86ebe635/csi-provisioner/
2026-03-14 09:55:30.432849 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-jqf62_60855808-2138-47e7-9539-df7c86ebe635/csi-provisioner/0.log
2026-03-14 09:55:30.432867 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-jqf62_60855808-2138-47e7-9539-df7c86ebe635/hostpath-provisioner/
2026-03-14 09:55:30.432884 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-jqf62_60855808-2138-47e7-9539-df7c86ebe635/hostpath-provisioner/0.log
2026-03-14 09:55:30.432901 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-jqf62_60855808-2138-47e7-9539-df7c86ebe635/liveness-probe/
2026-03-14 09:55:30.432917 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-jqf62_60855808-2138-47e7-9539-df7c86ebe635/liveness-probe/0.log
2026-03-14 09:55:30.432933 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-jqf62_60855808-2138-47e7-9539-df7c86ebe635/node-driver-registrar/
2026-03-14 09:55:30.432949 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-jqf62_60855808-2138-47e7-9539-df7c86ebe635/node-driver-registrar/0.log
2026-03-14 09:55:30.432969 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-7bb4cc7c98-pmz25_da0e2705-3765-4603-97ec-4ff1f5a2bf73/
2026-03-14 09:55:30.432985 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-7bb4cc7c98-pmz25_da0e2705-3765-4603-97ec-4ff1f5a2bf73/controller/
2026-03-14 09:55:30.433001 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-7bb4cc7c98-pmz25_da0e2705-3765-4603-97ec-4ff1f5a2bf73/controller/0.log.gz
2026-03-14 09:55:30.433108 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-7bb4cc7c98-pmz25_da0e2705-3765-4603-97ec-4ff1f5a2bf73/kube-rbac-proxy/
2026-03-14 09:55:30.433136 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-7bb4cc7c98-pmz25_da0e2705-3765-4603-97ec-4ff1f5a2bf73/kube-rbac-proxy/0.log
2026-03-14 09:55:30.433154 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/
2026-03-14 09:55:30.433170 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/383d64f7e8a816fadfef2c6f691a8b39260658483ae025d6241951d38865be9a.log
2026-03-14 09:55:30.433187 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/controller/
2026-03-14 09:55:30.433203 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/controller/0.log
2026-03-14 09:55:30.433219 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/cp-frr-files/
2026-03-14 09:55:30.433235 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/cp-frr-files/0.log
2026-03-14 09:55:30.433250 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/cp-metrics/
2026-03-14 09:55:30.433266 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/cp-metrics/0.log
2026-03-14 09:55:30.433281 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/cp-reloader/
2026-03-14 09:55:30.433296 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/cp-reloader/0.log
2026-03-14 09:55:30.433311 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/frr-metrics/
2026-03-14 09:55:30.433398 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/frr-metrics/0.log
2026-03-14 09:55:30.433425 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/frr/
2026-03-14 09:55:30.433442 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/frr/0.log.gz
2026-03-14 09:55:30.433459 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/kube-rbac-proxy-frr/
2026-03-14 09:55:30.433474 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/kube-rbac-proxy-frr/0.log
2026-03-14 09:55:30.433490 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/kube-rbac-proxy/
2026-03-14 09:55:30.433638 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/kube-rbac-proxy/0.log
2026-03-14 09:55:30.433666 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/reloader/
2026-03-14 09:55:30.433683 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-s58l8_a9d18aeb-d0f0-4312-945e-2eae3025fd59/reloader/0.log
2026-03-14 09:55:30.433700 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-webhook-server-bcc4b6f68-nd87q_1f771f81-34d4-4b3a-b2f9-1791c32f81fa/
2026-03-14 09:55:30.433716 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-webhook-server-bcc4b6f68-nd87q_1f771f81-34d4-4b3a-b2f9-1791c32f81fa/frr-k8s-webhook-server/
2026-03-14 09:55:30.433732 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-webhook-server-bcc4b6f68-nd87q_1f771f81-34d4-4b3a-b2f9-1791c32f81fa/frr-k8s-webhook-server/0.log
2026-03-14 09:55:30.433748 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-controller-manager-6ccbfd57fd-wcxtt_955961ad-ed3c-4fca-9da0-d9f44684fee8/
2026-03-14 09:55:30.433764 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-controller-manager-6ccbfd57fd-wcxtt_955961ad-ed3c-4fca-9da0-d9f44684fee8/manager/
2026-03-14 09:55:30.433779 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-controller-manager-6ccbfd57fd-wcxtt_955961ad-ed3c-4fca-9da0-d9f44684fee8/manager/0.log
2026-03-14 09:55:30.433795 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-webhook-server-886f6977-6gxlb_6ead3e16-6e7d-4598-993b-17d3fce555f3/
2026-03-14 09:55:30.433810 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-webhook-server-886f6977-6gxlb_6ead3e16-6e7d-4598-993b-17d3fce555f3/webhook-server/
2026-03-14 09:55:30.433826 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-webhook-server-886f6977-6gxlb_6ead3e16-6e7d-4598-993b-17d3fce555f3/webhook-server/0.log
2026-03-14 09:55:30.433842 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-npbfj_ce1b2f0a-e73a-4105-ba00-f91d243f6fd9/
2026-03-14 09:55:30.435570 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-npbfj_ce1b2f0a-e73a-4105-ba00-f91d243f6fd9/kube-rbac-proxy/
2026-03-14 09:55:30.435596 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-npbfj_ce1b2f0a-e73a-4105-ba00-f91d243f6fd9/kube-rbac-proxy/0.log
2026-03-14 09:55:30.435615 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-npbfj_ce1b2f0a-e73a-4105-ba00-f91d243f6fd9/speaker/
2026-03-14 09:55:30.435632 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-npbfj_ce1b2f0a-e73a-4105-ba00-f91d243f6fd9/speaker/0.log.gz
2026-03-14 09:55:30.435649 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver-operator_openshift-apiserver-operator-796bbdcf4f-4c865_f5b73e6d-b82a-48a3-8ae9-4afe284b4102/
2026-03-14 09:55:30.435665 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver-operator_openshift-apiserver-operator-796bbdcf4f-4c865_f5b73e6d-b82a-48a3-8ae9-4afe284b4102/openshift-apiserver-operator/
2026-03-14 09:55:30.435682 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver-operator_openshift-apiserver-operator-796bbdcf4f-4c865_f5b73e6d-b82a-48a3-8ae9-4afe284b4102/openshift-apiserver-operator/0.log
2026-03-14 09:55:30.435703 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-vcs4p_966098d4-f89c-4deb-a6e9-b1fae3316324/
2026-03-14 09:55:30.435719 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-vcs4p_966098d4-f89c-4deb-a6e9-b1fae3316324/fix-audit-permissions/
2026-03-14 09:55:30.435800 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-vcs4p_966098d4-f89c-4deb-a6e9-b1fae3316324/fix-audit-permissions/0.log
2026-03-14 09:55:30.435824 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-vcs4p_966098d4-f89c-4deb-a6e9-b1fae3316324/openshift-apiserver-check-endpoints/
2026-03-14 09:55:30.435842 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-vcs4p_966098d4-f89c-4deb-a6e9-b1fae3316324/openshift-apiserver-check-endpoints/0.log
2026-03-14 09:55:30.435859 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-vcs4p_966098d4-f89c-4deb-a6e9-b1fae3316324/openshift-apiserver/
2026-03-14 09:55:30.435876 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-vcs4p_966098d4-f89c-4deb-a6e9-b1fae3316324/openshift-apiserver/0.log
2026-03-14 09:55:30.436250 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication-operator_authentication-operator-69f744f599-7bkrj_f3953bf9-9497-4f03-a67e-e7d63bf1df9c/
2026-03-14 09:55:30.436274 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication-operator_authentication-operator-69f744f599-7bkrj_f3953bf9-9497-4f03-a67e-e7d63bf1df9c/authentication-operator/
2026-03-14 09:55:30.436292 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication-operator_authentication-operator-69f744f599-7bkrj_f3953bf9-9497-4f03-a67e-e7d63bf1df9c/authentication-operator/0.log
2026-03-14 09:55:30.436309 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication_oauth-openshift-7987bb8c7b-gzpwn_0f6b5a5b-c45b-4397-a29a-7424dd06c8ff/
2026-03-14 09:55:30.436325 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication_oauth-openshift-7987bb8c7b-gzpwn_0f6b5a5b-c45b-4397-a29a-7424dd06c8ff/oauth-openshift/
2026-03-14 09:55:30.436343 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication_oauth-openshift-7987bb8c7b-gzpwn_0f6b5a5b-c45b-4397-a29a-7424dd06c8ff/oauth-openshift/0.log
2026-03-14 09:55:30.436486 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-gq4nk_13a17846-77ac-4dba-a573-3b7d5c67da8d/
2026-03-14 09:55:30.436510 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-gq4nk_13a17846-77ac-4dba-a573-3b7d5c67da8d/kube-rbac-proxy/
2026-03-14 09:55:30.436527 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-gq4nk_13a17846-77ac-4dba-a573-3b7d5c67da8d/kube-rbac-proxy/0.log
2026-03-14 09:55:30.436544 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-gq4nk_13a17846-77ac-4dba-a573-3b7d5c67da8d/machine-approver-controller/
2026-03-14 09:55:30.436560 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-gq4nk_13a17846-77ac-4dba-a573-3b7d5c67da8d/machine-approver-controller/0.log
2026-03-14 09:55:30.436581 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-st9hs_4443557c-c15b-435f-80a8-44916ff7c31a/
2026-03-14 09:55:30.436597 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-st9hs_4443557c-c15b-435f-80a8-44916ff7c31a/ade586f97e06bcf0a55273569ef0da18306cb7bb5fbf24cb07b1d70be3710e1a.log
2026-03-14 09:55:30.436694 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-st9hs_4443557c-c15b-435f-80a8-44916ff7c31a/cluster-samples-operator-watch/
2026-03-14 09:55:30.436723 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-st9hs_4443557c-c15b-435f-80a8-44916ff7c31a/cluster-samples-operator-watch/0.log
2026-03-14 09:55:30.436741 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-st9hs_4443557c-c15b-435f-80a8-44916ff7c31a/cluster-samples-operator/
2026-03-14 09:55:30.436758 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-st9hs_4443557c-c15b-435f-80a8-44916ff7c31a/cluster-samples-operator/0.log
2026-03-14 09:55:30.436774 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-version_cluster-version-operator-5c965bbfc6-mwsr6_25e45193-cbd7-4d19-be68-38e0008a5a3b/
2026-03-14 09:55:30.436791 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-version_cluster-version-operator-5c965bbfc6-mwsr6_25e45193-cbd7-4d19-be68-38e0008a5a3b/cluster-version-operator/
2026-03-14 09:55:30.436806 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-version_cluster-version-operator-5c965bbfc6-mwsr6_25e45193-cbd7-4d19-be68-38e0008a5a3b/cluster-version-operator/0.log.gz
2026-03-14 09:55:30.436822 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-6bzr5_9ee8610e-b521-4ee6-8dda-01c0aa50ae3e/
2026-03-14 09:55:30.436838 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-6bzr5_9ee8610e-b521-4ee6-8dda-01c0aa50ae3e/openshift-api/
2026-03-14 09:55:30.436853 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-6bzr5_9ee8610e-b521-4ee6-8dda-01c0aa50ae3e/openshift-api/0.log
2026-03-14 09:55:30.436869 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-6bzr5_9ee8610e-b521-4ee6-8dda-01c0aa50ae3e/openshift-config-operator/
2026-03-14 09:55:30.436884 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-6bzr5_9ee8610e-b521-4ee6-8dda-01c0aa50ae3e/openshift-config-operator/0.log
2026-03-14 09:55:30.436899 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console-operator_console-operator-58897d9998-d9kk5_e7fa9107-c6f8-4ecd-bb56-62b7be869ec9/
2026-03-14 09:55:30.436915 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console-operator_console-operator-58897d9998-d9kk5_e7fa9107-c6f8-4ecd-bb56-62b7be869ec9/console-operator/
2026-03-14 09:55:30.436930 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console-operator_console-operator-58897d9998-d9kk5_e7fa9107-c6f8-4ecd-bb56-62b7be869ec9/console-operator/0.log
2026-03-14 09:55:30.436945 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_console-699f6c78c-n8rhg_8c4a5b86-919c-4db1-bb99-c3f1699bb9a9/
2026-03-14 09:55:30.436968 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_console-699f6c78c-n8rhg_8c4a5b86-919c-4db1-bb99-c3f1699bb9a9/console/
2026-03-14 09:55:30.436985 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_console-699f6c78c-n8rhg_8c4a5b86-919c-4db1-bb99-c3f1699bb9a9/console/0.log
2026-03-14 09:55:30.437000 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_downloads-7954f5f757-lpkrv_343e0673-aad7-49c1-91b7-f5fd88579db3/
2026-03-14 09:55:30.437403 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_downloads-7954f5f757-lpkrv_343e0673-aad7-49c1-91b7-f5fd88579db3/download-server/
2026-03-14 09:55:30.437435 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_downloads-7954f5f757-lpkrv_343e0673-aad7-49c1-91b7-f5fd88579db3/download-server/0.log
2026-03-14 09:55:30.437454 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager-operator_openshift-controller-manager-operator-756b6f6bc6-jqqmv_44ffbc86-37af-4fa5-bc0c-58084b961597/
2026-03-14 09:55:30.437471 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager-operator_openshift-controller-manager-operator-756b6f6bc6-jqqmv_44ffbc86-37af-4fa5-bc0c-58084b961597/openshift-controller-manager-operator/
2026-03-14 09:55:30.437488 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager-operator_openshift-controller-manager-operator-756b6f6bc6-jqqmv_44ffbc86-37af-4fa5-bc0c-58084b961597/openshift-controller-manager-operator/0.log
2026-03-14 09:55:30.437504 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager_controller-manager-84f99cfc54-5kv5k_7eb13c2f-e8ec-4f7c-8b17-e6d425c7c7ca/
2026-03-14 09:55:30.437521 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager_controller-manager-84f99cfc54-5kv5k_7eb13c2f-e8ec-4f7c-8b17-e6d425c7c7ca/controller-manager/
2026-03-14 09:55:30.437537 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager_controller-manager-84f99cfc54-5kv5k_7eb13c2f-e8ec-4f7c-8b17-e6d425c7c7ca/controller-manager/0.log
2026-03-14 09:55:30.437553 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-776c5_9231d09e-fc87-40bf-85fe-c1d16e3ee943/
2026-03-14 09:55:30.437568 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-776c5_9231d09e-fc87-40bf-85fe-c1d16e3ee943/dns-operator/
2026-03-14 09:55:30.437584 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-776c5_9231d09e-fc87-40bf-85fe-c1d16e3ee943/dns-operator/0.log
2026-03-14 09:55:30.437599 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-776c5_9231d09e-fc87-40bf-85fe-c1d16e3ee943/kube-rbac-proxy/
2026-03-14 09:55:30.437615 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-776c5_9231d09e-fc87-40bf-85fe-c1d16e3ee943/kube-rbac-proxy/0.log
2026-03-14 09:55:30.437631 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-m5sxz_93ee50e1-8448-4b8e-89be-bba3e87c0843/
2026-03-14 09:55:30.437754 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-m5sxz_93ee50e1-8448-4b8e-89be-bba3e87c0843/dns/
2026-03-14 09:55:30.437776 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-m5sxz_93ee50e1-8448-4b8e-89be-bba3e87c0843/dns/0.log
2026-03-14 09:55:30.437799 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-m5sxz_93ee50e1-8448-4b8e-89be-bba3e87c0843/kube-rbac-proxy/
2026-03-14 09:55:30.437816 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-m5sxz_93ee50e1-8448-4b8e-89be-bba3e87c0843/kube-rbac-proxy/0.log
2026-03-14 09:55:30.437831 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_node-resolver-h264v_d7e96975-9562-47ec-9476-e593f8d6be98/
2026-03-14 09:55:30.437847 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_node-resolver-h264v_d7e96975-9562-47ec-9476-e593f8d6be98/dns-node-resolver/
2026-03-14 09:55:30.437863 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_node-resolver-h264v_d7e96975-9562-47ec-9476-e593f8d6be98/dns-node-resolver/0.log
2026-03-14 09:55:30.437878 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd-operator_etcd-operator-b45778765-j9qjz_b3520644-d9b3-41e1-8293-7bc8529ffbca/
2026-03-14 09:55:30.437894 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd-operator_etcd-operator-b45778765-j9qjz_b3520644-d9b3-41e1-8293-7bc8529ffbca/etcd-operator/
2026-03-14 09:55:30.437909 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd-operator_etcd-operator-b45778765-j9qjz_b3520644-d9b3-41e1-8293-7bc8529ffbca/etcd-operator/0.log
2026-03-14 09:55:30.437925 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/
2026-03-14 09:55:30.437941 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-ensure-env-vars/
2026-03-14 09:55:30.438113 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-ensure-env-vars/0.log
2026-03-14 09:55:30.438137 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-metrics/
2026-03-14 09:55:30.438154 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-metrics/0.log
2026-03-14 09:55:30.438170 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-readyz/
2026-03-14 09:55:30.438186 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-readyz/0.log
2026-03-14 09:55:30.438201 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-resources-copy/
2026-03-14 09:55:30.438217 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-resources-copy/0.log
2026-03-14 09:55:30.438233 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-rev/
2026-03-14 09:55:30.438249 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-rev/0.log
2026-03-14 09:55:30.438265 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd/
2026-03-14 09:55:30.438282 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd/0.log
2026-03-14 09:55:30.438299 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcdctl/
2026-03-14 09:55:30.438320 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcdctl/0.log
2026-03-14 09:55:30.438588 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/setup/
2026-03-14 09:55:30.438614 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/setup/0.log
2026-03-14 09:55:30.438633 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_cluster-image-registry-operator-dc59b4c8b-c8m6n_4708c5fe-cd4d-48ec-9cca-2c6f89c2fd8f/
2026-03-14 09:55:30.438650 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_cluster-image-registry-operator-dc59b4c8b-c8m6n_4708c5fe-cd4d-48ec-9cca-2c6f89c2fd8f/cluster-image-registry-operator/
2026-03-14 09:55:30.438668 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_cluster-image-registry-operator-dc59b4c8b-c8m6n_4708c5fe-cd4d-48ec-9cca-2c6f89c2fd8f/cluster-image-registry-operator/0.log
2026-03-14 09:55:30.438684 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_image-registry-66df7c8f76-s8rkh_a77d4adc-529a-4d32-93d9-ef0ee376a4ec/
2026-03-14 09:55:30.438700 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_image-registry-66df7c8f76-s8rkh_a77d4adc-529a-4d32-93d9-ef0ee376a4ec/registry/
2026-03-14 09:55:30.438716 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_image-registry-66df7c8f76-s8rkh_a77d4adc-529a-4d32-93d9-ef0ee376a4ec/registry/0.log
2026-03-14 09:55:30.438732 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_node-ca-gqfh2_7535708d-1eec-4d4c-b0eb-f5343af71b3d/
2026-03-14 09:55:30.438747 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_node-ca-gqfh2_7535708d-1eec-4d4c-b0eb-f5343af71b3d/node-ca/
2026-03-14 09:55:30.438768 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_node-ca-gqfh2_7535708d-1eec-4d4c-b0eb-f5343af71b3d/node-ca/0.log
2026-03-14 09:55:30.438890 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558024-slxr8_876cec90-17ca-4a4c-8cbc-462048c32b1c/
2026-03-14 09:55:30.438922 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558024-slxr8_876cec90-17ca-4a4c-8cbc-462048c32b1c/oc/
2026-03-14 09:55:30.438944 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558024-slxr8_876cec90-17ca-4a4c-8cbc-462048c32b1c/oc/0.log
2026-03-14 09:55:30.438966 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558026-9c6gt_864a9dee-9abe-458e-9fc7-151e1df7c41c/
2026-03-14 09:55:30.438986 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558026-9c6gt_864a9dee-9abe-458e-9fc7-151e1df7c41c/oc/
2026-03-14 09:55:30.439007 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558026-9c6gt_864a9dee-9abe-458e-9fc7-151e1df7c41c/oc/0.log
2026-03-14 09:55:30.439043 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558028-dcwt6_9435121f-6fad-4634-9595-b998a87504a2/
2026-03-14 09:55:30.439065 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558028-dcwt6_9435121f-6fad-4634-9595-b998a87504a2/oc/
2026-03-14 09:55:30.439085 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558028-dcwt6_9435121f-6fad-4634-9595-b998a87504a2/oc/0.log
2026-03-14 09:55:30.439305 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558030-d7hqz_fa4ab338-ea51-4535-8ab4-4e7f8bf025e4/
2026-03-14 09:55:30.439338 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558030-d7hqz_fa4ab338-ea51-4535-8ab4-4e7f8bf025e4/oc/
2026-03-14 09:55:30.439356 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29558030-d7hqz_fa4ab338-ea51-4535-8ab4-4e7f8bf025e4/oc/0.log
2026-03-14 09:55:30.439372 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-canary_ingress-canary-xm4x6_59da700a-d9ae-49ac-a504-b7d9ea7ff0b6/
2026-03-14 09:55:30.439388 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-canary_ingress-canary-xm4x6_59da700a-d9ae-49ac-a504-b7d9ea7ff0b6/serve-healthcheck-canary/
2026-03-14 09:55:30.439404 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-canary_ingress-canary-xm4x6_59da700a-d9ae-49ac-a504-b7d9ea7ff0b6/serve-healthcheck-canary/0.log
2026-03-14 09:55:30.439420 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-jspq2_61c89047-11ed-4a9e-b861-25e144d42b5e/
2026-03-14 09:55:30.439436 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-jspq2_61c89047-11ed-4a9e-b861-25e144d42b5e/ingress-operator/
2026-03-14 09:55:30.439451 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-jspq2_61c89047-11ed-4a9e-b861-25e144d42b5e/ingress-operator/0.log
2026-03-14 09:55:30.439467 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-jspq2_61c89047-11ed-4a9e-b861-25e144d42b5e/kube-rbac-proxy/
2026-03-14 09:55:30.439483 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-jspq2_61c89047-11ed-4a9e-b861-25e144d42b5e/kube-rbac-proxy/0.log
2026-03-14 09:55:30.439498 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress_router-default-5444994796-hsfss_16733c4e-be2e-4b5a-885c-6d2fab583caf/
2026-03-14 09:55:30.439514 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress_router-default-5444994796-hsfss_16733c4e-be2e-4b5a-885c-6d2fab583caf/router/
2026-03-14 09:55:30.439529 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress_router-default-5444994796-hsfss_16733c4e-be2e-4b5a-885c-6d2fab583caf/router/0.log
2026-03-14 09:55:30.439746 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver-operator_kube-apiserver-operator-766d6c64bb-s5hxg_236467b8-ffbf-4e32-ba1c-b188938f8ff5/
2026-03-14 09:55:30.439771 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver-operator_kube-apiserver-operator-766d6c64bb-s5hxg_236467b8-ffbf-4e32-ba1c-b188938f8ff5/kube-apiserver-operator/
2026-03-14 09:55:30.439788 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver-operator_kube-apiserver-operator-766d6c64bb-s5hxg_236467b8-ffbf-4e32-ba1c-b188938f8ff5/kube-apiserver-operator/0.log
2026-03-14 09:55:30.439804 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_installer-9-crc_ab17437b-05d0-4ae8-8d53-2b7eb384acbe/
2026-03-14 09:55:30.439820 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_installer-9-crc_ab17437b-05d0-4ae8-8d53-2b7eb384acbe/installer/
2026-03-14 09:55:30.439840 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_installer-9-crc_ab17437b-05d0-4ae8-8d53-2b7eb384acbe/installer/0.log
2026-03-14 09:55:30.439857 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/
2026-03-14 09:55:30.439873 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-cert-regeneration-controller/
2026-03-14 09:55:30.439888 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-cert-regeneration-controller/0.log
2026-03-14 09:55:30.439904 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-cert-syncer/
2026-03-14 09:55:30.439920 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-cert-syncer/0.log
2026-03-14 09:55:30.439935 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-check-endpoints/
2026-03-14 09:55:30.440085 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-check-endpoints/0.log
2026-03-14 09:55:30.440111 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-insecure-readyz/
2026-03-14 09:55:30.440129 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-insecure-readyz/0.log
2026-03-14 09:55:30.440144 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver/
2026-03-14 09:55:30.440160 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver/0.log
2026-03-14 09:55:30.440175 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/setup/
2026-03-14 09:55:30.440191 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/setup/0.log
2026-03-14 09:55:30.440207 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-8-crc_fc199c10-e8a5-409a-9ca5-b99a872682c2/
2026-03-14 09:55:30.440222 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-8-crc_fc199c10-e8a5-409a-9ca5-b99a872682c2/pruner/
2026-03-14 09:55:30.440238 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-8-crc_fc199c10-e8a5-409a-9ca5-b99a872682c2/pruner/0.log
2026-03-14 09:55:30.440253 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-9-crc_feea3874-1da5-4b39-b76d-06eea186b678/
2026-03-14 09:55:30.440269 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-9-crc_feea3874-1da5-4b39-b76d-06eea186b678/pruner/
2026-03-14 09:55:30.440284 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-9-crc_feea3874-1da5-4b39-b76d-06eea186b678/pruner/0.log
2026-03-14 09:55:30.440411 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager-operator_kube-controller-manager-operator-78b949d7b-ksv2k_cf9fc759-1702-415b-8879-0f5138ad1b46/
2026-03-14 09:55:30.440436 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager-operator_kube-controller-manager-operator-78b949d7b-ksv2k_cf9fc759-1702-415b-8879-0f5138ad1b46/kube-controller-manager-operator/
2026-03-14 09:55:30.440454 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager-operator_kube-controller-manager-operator-78b949d7b-ksv2k_cf9fc759-1702-415b-8879-0f5138ad1b46/kube-controller-manager-operator/0.log
2026-03-14 09:55:30.440470 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/
2026-03-14 09:55:30.440486 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/cluster-policy-controller/
2026-03-14 09:55:30.440502 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/cluster-policy-controller/1.log
2026-03-14 09:55:30.440519 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/cluster-policy-controller/2.log
2026-03-14 09:55:30.440535 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager-cert-syncer/
2026-03-14 09:55:30.440551 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager-cert-syncer/0.log
2026-03-14 09:55:30.440568 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager-recovery-controller/
2026-03-14 09:55:30.440583 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager-recovery-controller/0.log
2026-03-14 09:55:30.440600 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager/
2026-03-14 09:55:30.440714 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager/0.log
2026-03-14 09:55:30.440739 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager/1.log.gz
2026-03-14 09:55:30.440756 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_revision-pruner-9-crc_c5f43a71-8ea3-46f5-a6ce-e1ffaff74a79/
2026-03-14 09:55:30.440773 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_revision-pruner-9-crc_c5f43a71-8ea3-46f5-a6ce-e1ffaff74a79/pruner/
2026-03-14 09:55:30.440789 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_revision-pruner-9-crc_c5f43a71-8ea3-46f5-a6ce-e1ffaff74a79/pruner/0.log
2026-03-14 09:55:30.440805 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler-operator_openshift-kube-scheduler-operator-5fdd9b5758-pftg4_b616bb02-0180-4af3-922a-6a09d2da3d67/
2026-03-14 09:55:30.440825 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler-operator_openshift-kube-scheduler-operator-5fdd9b5758-pftg4_b616bb02-0180-4af3-922a-6a09d2da3d67/kube-scheduler-operator-container/
2026-03-14 09:55:30.440842 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler-operator_openshift-kube-scheduler-operator-5fdd9b5758-pftg4_b616bb02-0180-4af3-922a-6a09d2da3d67/kube-scheduler-operator-container/0.log
2026-03-14 09:55:30.440859 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/
2026-03-14 09:55:30.440875 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler-cert-syncer/
2026-03-14 09:55:30.440892 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler-cert-syncer/0.log
2026-03-14 09:55:30.440908 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler-recovery-controller/
2026-03-14 09:55:30.440923 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler-recovery-controller/0.log
2026-03-14 09:55:30.440939 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler/
2026-03-14 09:55:30.441065 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler/0.log
2026-03-14 09:55:30.441098 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/wait-for-host-port/
2026-03-14 09:55:30.441116 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/wait-for-host-port/0.log
2026-03-14 09:55:30.441133 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-b67b599dd-8hw77_7f915e7e-5c5a-4b5c-8b46-459a5bddb130/
2026-03-14 09:55:30.441150 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-b67b599dd-8hw77_7f915e7e-5c5a-4b5c-8b46-459a5bddb130/kube-storage-version-migrator-operator/
2026-03-14 09:55:30.441167 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-b67b599dd-8hw77_7f915e7e-5c5a-4b5c-8b46-459a5bddb130/kube-storage-version-migrator-operator/0.log
2026-03-14 09:55:30.441183 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-tb6lx_c0e5952a-041e-4372-b077-b13baabaabd0/
2026-03-14 09:55:30.441199 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-tb6lx_c0e5952a-041e-4372-b077-b13baabaabd0/graceful-termination/
2026-03-14 09:55:30.441215 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-tb6lx_c0e5952a-041e-4372-b077-b13baabaabd0/graceful-termination/0.log
2026-03-14 09:55:30.441235 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-tb6lx_c0e5952a-041e-4372-b077-b13baabaabd0/migrator/
2026-03-14 09:55:30.441251 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-tb6lx_c0e5952a-041e-4372-b077-b13baabaabd0/migrator/0.log
2026-03-14 09:55:30.441267 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_control-plane-machine-set-operator-78cbb6b69f-d7wqv_1666768a-cdbb-4ab1-83d8-b1ad0444f167/
2026-03-14 09:55:30.441283 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_control-plane-machine-set-operator-78cbb6b69f-d7wqv_1666768a-cdbb-4ab1-83d8-b1ad0444f167/control-plane-machine-set-operator/
2026-03-14 09:55:30.441299 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_control-plane-machine-set-operator-78cbb6b69f-d7wqv_1666768a-cdbb-4ab1-83d8-b1ad0444f167/control-plane-machine-set-operator/0.log
2026-03-14 09:55:30.441424 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-8gwm6_095068b1-bf13-43a2-a250-a0eaeb60c6ae/
2026-03-14 09:55:30.441446 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-8gwm6_095068b1-bf13-43a2-a250-a0eaeb60c6ae/kube-rbac-proxy/
2026-03-14 09:55:30.441462 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-8gwm6_095068b1-bf13-43a2-a250-a0eaeb60c6ae/kube-rbac-proxy/0.log
2026-03-14 09:55:30.441478 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-8gwm6_095068b1-bf13-43a2-a250-a0eaeb60c6ae/machine-api-operator/
2026-03-14 09:55:30.441494 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-8gwm6_095068b1-bf13-43a2-a250-a0eaeb60c6ae/machine-api-operator/0.log
2026-03-14 09:55:30.441614 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/
2026-03-14 09:55:30.441643 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/kube-rbac-proxy-crio/
2026-03-14 09:55:30.441661 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/kube-rbac-proxy-crio/0.log
2026-03-14 09:55:30.441677 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/setup/
2026-03-14 09:55:30.441693 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/setup/0.log
2026-03-14 09:55:30.441709 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-nvgzt_a78cde11-057e-4d0d-986e-bcda5f4964d6/
2026-03-14 09:55:30.441724 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-nvgzt_a78cde11-057e-4d0d-986e-bcda5f4964d6/kube-rbac-proxy/
2026-03-14 09:55:30.441857 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-nvgzt_a78cde11-057e-4d0d-986e-bcda5f4964d6/kube-rbac-proxy/0.log
2026-03-14 09:55:30.441879 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-nvgzt_a78cde11-057e-4d0d-986e-bcda5f4964d6/machine-config-controller/
2026-03-14 09:55:30.441901 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-nvgzt_a78cde11-057e-4d0d-986e-bcda5f4964d6/machine-config-controller/0.log
2026-03-14 09:55:30.441917 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-mxjrk_9ba20367-e506-422e-a846-eb1525cb3b94/
2026-03-14 09:55:30.442059 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-mxjrk_9ba20367-e506-422e-a846-eb1525cb3b94/kube-rbac-proxy/
2026-03-14 09:55:30.442091 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-mxjrk_9ba20367-e506-422e-a846-eb1525cb3b94/kube-rbac-proxy/0.log
2026-03-14 09:55:30.442109 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-mxjrk_9ba20367-e506-422e-a846-eb1525cb3b94/machine-config-daemon/
2026-03-14 09:55:30.442126 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-mxjrk_9ba20367-e506-422e-a846-eb1525cb3b94/machine-config-daemon/11.log
2026-03-14 09:55:30.442142 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-sw2ht_c30cdb70-a97a-4750-8a55-e4aaa0e44f9b/
2026-03-14 09:55:30.442157 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-sw2ht_c30cdb70-a97a-4750-8a55-e4aaa0e44f9b/kube-rbac-proxy/
2026-03-14 09:55:30.442281 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-sw2ht_c30cdb70-a97a-4750-8a55-e4aaa0e44f9b/kube-rbac-proxy/0.log
2026-03-14 09:55:30.442303 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-sw2ht_c30cdb70-a97a-4750-8a55-e4aaa0e44f9b/machine-config-operator/
2026-03-14 09:55:30.442320 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-sw2ht_c30cdb70-a97a-4750-8a55-e4aaa0e44f9b/machine-config-operator/0.log
2026-03-14 09:55:30.442337 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-server-d7cv4_8f87de9e-e590-4bb0-8f62-a5f4bf1922cb/
2026-03-14 09:55:30.442352 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-server-d7cv4_8f87de9e-e590-4bb0-8f62-a5f4bf1922cb/machine-config-server/
2026-03-14 09:55:30.442368 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-server-d7cv4_8f87de9e-e590-4bb0-8f62-a5f4bf1922cb/machine-config-server/0.log
2026-03-14 09:55:30.442486 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde8744sb7r_75426cf5-0280-405c-a216-58ba481acb46/
2026-03-14 09:55:30.442512 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde8744sb7r_75426cf5-0280-405c-a216-58ba481acb46/extract/
2026-03-14 09:55:30.442530 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde8744sb7r_75426cf5-0280-405c-a216-58ba481acb46/extract/0.log
2026-03-14 09:55:30.442551 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde8744sb7r_75426cf5-0280-405c-a216-58ba481acb46/pull/
2026-03-14 09:55:30.442567 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde8744sb7r_75426cf5-0280-405c-a216-58ba481acb46/pull/0.log
2026-03-14 09:55:30.442756 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde8744sb7r_75426cf5-0280-405c-a216-58ba481acb46/util/
2026-03-14 09:55:30.442778 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde8744sb7r_75426cf5-0280-405c-a216-58ba481acb46/util/0.log
2026-03-14 09:55:30.442795 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c1tlsz6_37548b9f-0521-4b58-a42a-a023fe66022b/
2026-03-14 09:55:30.442812 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c1tlsz6_37548b9f-0521-4b58-a42a-a023fe66022b/extract/
2026-03-14 09:55:30.442828 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c1tlsz6_37548b9f-0521-4b58-a42a-a023fe66022b/extract/0.log
2026-03-14 09:55:30.442973 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c1tlsz6_37548b9f-0521-4b58-a42a-a023fe66022b/pull/
2026-03-14 09:55:30.442997 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c1tlsz6_37548b9f-0521-4b58-a42a-a023fe66022b/pull/0.log
2026-03-14 09:55:30.443033 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c1tlsz6_37548b9f-0521-4b58-a42a-a023fe66022b/util/
2026-03-14 09:55:30.443165 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c1tlsz6_37548b9f-0521-4b58-a42a-a023fe66022b/util/0.log
2026-03-14 09:55:30.443194 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e57vqd9_a710d73a-4449-44aa-b6e0-198adf069d08/
2026-03-14 09:55:30.443213 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e57vqd9_a710d73a-4449-44aa-b6e0-198adf069d08/extract/
2026-03-14 09:55:30.443229 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e57vqd9_a710d73a-4449-44aa-b6e0-198adf069d08/extract/0.log
2026-03-14 09:55:30.443246 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e57vqd9_a710d73a-4449-44aa-b6e0-198adf069d08/pull/
2026-03-14 09:55:30.443263 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e57vqd9_a710d73a-4449-44aa-b6e0-198adf069d08/pull/0.log
2026-03-14 09:55:30.443279 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e57vqd9_a710d73a-4449-44aa-b6e0-198adf069d08/util/
2026-03-14 09:55:30.443295 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e57vqd9_a710d73a-4449-44aa-b6e0-198adf069d08/util/0.log
2026-03-14 09:55:30.443316 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f08spsnl_804193a1-2db0-4ac9-a126-4c79735f8302/
2026-03-14 09:55:30.443333 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f08spsnl_804193a1-2db0-4ac9-a126-4c79735f8302/extract/
2026-03-14 09:55:30.443383 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f08spsnl_804193a1-2db0-4ac9-a126-4c79735f8302/extract/0.log
2026-03-14 09:55:30.443405 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f08spsnl_804193a1-2db0-4ac9-a126-4c79735f8302/pull/
2026-03-14 09:55:30.443422 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f08spsnl_804193a1-2db0-4ac9-a126-4c79735f8302/pull/0.log
2026-03-14 09:55:30.443438 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f08spsnl_804193a1-2db0-4ac9-a126-4c79735f8302/util/
2026-03-14 09:55:30.443567 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f08spsnl_804193a1-2db0-4ac9-a126-4c79735f8302/util/0.log
2026-03-14 09:55:30.443592 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-vzmj4_7e5025eb-1d48-4be9-ab21-5aa5c0436d7b/
2026-03-14 09:55:30.443610 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-vzmj4_7e5025eb-1d48-4be9-ab21-5aa5c0436d7b/extract-content/
2026-03-14 09:55:30.443626 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-vzmj4_7e5025eb-1d48-4be9-ab21-5aa5c0436d7b/extract-content/0.log
2026-03-14 09:55:30.443642 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-vzmj4_7e5025eb-1d48-4be9-ab21-5aa5c0436d7b/extract-utilities/
2026-03-14 09:55:30.443658 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-vzmj4_7e5025eb-1d48-4be9-ab21-5aa5c0436d7b/extract-utilities/0.log
2026-03-14 09:55:30.443673 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-vzmj4_7e5025eb-1d48-4be9-ab21-5aa5c0436d7b/registry-server/
2026-03-14 09:55:30.443838 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-vzmj4_7e5025eb-1d48-4be9-ab21-5aa5c0436d7b/registry-server/0.log.gz
2026-03-14 09:55:30.443862 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-fnwsk_0eb1f1fb-17bc-4cc7-a158-6576b474e996/
2026-03-14 09:55:30.443879 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-fnwsk_0eb1f1fb-17bc-4cc7-a158-6576b474e996/extract-content/
2026-03-14 09:55:30.443896 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-fnwsk_0eb1f1fb-17bc-4cc7-a158-6576b474e996/extract-content/0.log
2026-03-14 09:55:30.443911 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-fnwsk_0eb1f1fb-17bc-4cc7-a158-6576b474e996/extract-utilities/
2026-03-14 09:55:30.443931 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-fnwsk_0eb1f1fb-17bc-4cc7-a158-6576b474e996/extract-utilities/0.log
2026-03-14 09:55:30.443947 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-fnwsk_0eb1f1fb-17bc-4cc7-a158-6576b474e996/registry-server/
2026-03-14 09:55:30.444051 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-fnwsk_0eb1f1fb-17bc-4cc7-a158-6576b474e996/registry-server/0.log.gz
2026-03-14 09:55:30.444083 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_marketplace-operator-79b997595-7gnpp_92ce0359-87bc-46d6-8673-b10febbf0742/
2026-03-14 09:55:30.444102 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_marketplace-operator-79b997595-7gnpp_92ce0359-87bc-46d6-8673-b10febbf0742/marketplace-operator/
2026-03-14 09:55:30.444119 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_marketplace-operator-79b997595-7gnpp_92ce0359-87bc-46d6-8673-b10febbf0742/marketplace-operator/0.log
2026-03-14 09:55:30.444135 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-6mtml_098e20c5-7d99-4468-896d-fb13222a450b/
2026-03-14 09:55:30.444151 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-6mtml_098e20c5-7d99-4468-896d-fb13222a450b/extract-content/
2026-03-14 09:55:30.444167 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-6mtml_098e20c5-7d99-4468-896d-fb13222a450b/extract-content/0.log
2026-03-14 09:55:30.444372 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-6mtml_098e20c5-7d99-4468-896d-fb13222a450b/extract-utilities/
2026-03-14 09:55:30.444397 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-6mtml_098e20c5-7d99-4468-896d-fb13222a450b/extract-utilities/0.log
2026-03-14 09:55:30.444415 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-6mtml_098e20c5-7d99-4468-896d-fb13222a450b/registry-server/
2026-03-14 09:55:30.444431 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-6mtml_098e20c5-7d99-4468-896d-fb13222a450b/registry-server/0.log
2026-03-14 09:55:30.444446 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-5d92s_e7f24159-2581-4653-be69-3aae1dd7e3f9/
2026-03-14 09:55:30.444462 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-5d92s_e7f24159-2581-4653-be69-3aae1dd7e3f9/extract-content/
2026-03-14 09:55:30.444478 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-5d92s_e7f24159-2581-4653-be69-3aae1dd7e3f9/extract-content/0.log
2026-03-14 09:55:30.444605 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-5d92s_e7f24159-2581-4653-be69-3aae1dd7e3f9/extract-utilities/
2026-03-14 09:55:30.444631 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-5d92s_e7f24159-2581-4653-be69-3aae1dd7e3f9/extract-utilities/0.log
2026-03-14 09:55:30.444648 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-5d92s_e7f24159-2581-4653-be69-3aae1dd7e3f9/registry-server/
2026-03-14 09:55:30.444664 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-5d92s_e7f24159-2581-4653-be69-3aae1dd7e3f9/registry-server/0.log.gz
2026-03-14 09:55:30.444685 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-v95z4_f6520c5a-bc4e-48e8-a9a1-3738103894a1/
2026-03-14 09:55:30.444741 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-v95z4_f6520c5a-bc4e-48e8-a9a1-3738103894a1/extract-content/
2026-03-14 09:55:30.444848 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-v95z4_f6520c5a-bc4e-48e8-a9a1-3738103894a1/extract-content/0.log
2026-03-14 09:55:30.444874 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-v95z4_f6520c5a-bc4e-48e8-a9a1-3738103894a1/extract-utilities/
2026-03-14 09:55:30.444891 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-v95z4_f6520c5a-bc4e-48e8-a9a1-3738103894a1/extract-utilities/0.log
2026-03-14 09:55:30.444907 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-5xlw2_b2131e0d-5d1b-4913-8908-e15859b063a4/
2026-03-14 09:55:30.444923 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-5xlw2_b2131e0d-5d1b-4913-8908-e15859b063a4/bond-cni-plugin/
2026-03-14 09:55:30.444939 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-5xlw2_b2131e0d-5d1b-4913-8908-e15859b063a4/bond-cni-plugin/0.log
2026-03-14 09:55:30.445052 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-5xlw2_b2131e0d-5d1b-4913-8908-e15859b063a4/cni-plugins/
2026-03-14 09:55:30.445078 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-5xlw2_b2131e0d-5d1b-4913-8908-e15859b063a4/cni-plugins/0.log
2026-03-14 09:55:30.445096 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-5xlw2_b2131e0d-5d1b-4913-8908-e15859b063a4/egress-router-binary-copy/
2026-03-14 09:55:30.445112 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-5xlw2_b2131e0d-5d1b-4913-8908-e15859b063a4/egress-router-binary-copy/0.log
2026-03-14 09:55:30.445128 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-5xlw2_b2131e0d-5d1b-4913-8908-e15859b063a4/kube-multus-additional-cni-plugins/
2026-03-14 09:55:30.445143 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-5xlw2_b2131e0d-5d1b-4913-8908-e15859b063a4/kube-multus-additional-cni-plugins/0.log
2026-03-14 09:55:30.445159 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-5xlw2_b2131e0d-5d1b-4913-8908-e15859b063a4/routeoverride-cni/
2026-03-14 09:55:30.445278 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-5xlw2_b2131e0d-5d1b-4913-8908-e15859b063a4/routeoverride-cni/0.log
2026-03-14 09:55:30.445302 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-5xlw2_b2131e0d-5d1b-4913-8908-e15859b063a4/whereabouts-cni-bincopy/
2026-03-14 09:55:30.445320 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-5xlw2_b2131e0d-5d1b-4913-8908-e15859b063a4/whereabouts-cni-bincopy/0.log
2026-03-14 09:55:30.445336 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-5xlw2_b2131e0d-5d1b-4913-8908-e15859b063a4/whereabouts-cni/
2026-03-14 09:55:30.445357 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-5xlw2_b2131e0d-5d1b-4913-8908-e15859b063a4/whereabouts-cni/0.log
2026-03-14 09:55:30.445373 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-mqkjv_1e01187b-7ff4-40bd-b726-33eedbc33f3c/
2026-03-14 09:55:30.445522 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-mqkjv_1e01187b-7ff4-40bd-b726-33eedbc33f3c/kube-rbac-proxy/
2026-03-14 09:55:30.445543 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-mqkjv_1e01187b-7ff4-40bd-b726-33eedbc33f3c/kube-rbac-proxy/0.log
2026-03-14 09:55:30.445560 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-mqkjv_1e01187b-7ff4-40bd-b726-33eedbc33f3c/multus-admission-controller/
2026-03-14 09:55:30.445577 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-mqkjv_1e01187b-7ff4-40bd-b726-33eedbc33f3c/multus-admission-controller/0.log
2026-03-14 09:55:30.445822 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-sgnxb_7528e098-09d4-436f-a32d-a0e82e76b8e0/
2026-03-14 09:55:30.446865 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-sgnxb_7528e098-09d4-436f-a32d-a0e82e76b8e0/kube-multus/
2026-03-14 09:55:30.446891 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-sgnxb_7528e098-09d4-436f-a32d-a0e82e76b8e0/kube-multus/1.log
2026-03-14 09:55:30.446910 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-sgnxb_7528e098-09d4-436f-a32d-a0e82e76b8e0/kube-multus/2.log
2026-03-14 09:55:30.446928 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-42pn5_bf6ad235-d99c-46a7-8c2d-6fc12fc07c10/
2026-03-14 09:55:30.447051 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-42pn5_bf6ad235-d99c-46a7-8c2d-6fc12fc07c10/kube-rbac-proxy/
2026-03-14 09:55:30.447078 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-42pn5_bf6ad235-d99c-46a7-8c2d-6fc12fc07c10/kube-rbac-proxy/0.log
2026-03-14 09:55:30.447108 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-42pn5_bf6ad235-d99c-46a7-8c2d-6fc12fc07c10/network-metrics-daemon/
2026-03-14 09:55:30.447126 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-42pn5_bf6ad235-d99c-46a7-8c2d-6fc12fc07c10/network-metrics-daemon/0.log
2026-03-14 09:55:30.447253 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-console_networking-console-plugin-85b44fc459-gdk6g_5fe485a1-e14f-4c09-b5b9-f252bc42b7e8/
2026-03-14 09:55:30.447275 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-console_networking-console-plugin-85b44fc459-gdk6g_5fe485a1-e14f-4c09-b5b9-f252bc42b7e8/networking-console-plugin/
2026-03-14 09:55:30.447293 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-console_networking-console-plugin-85b44fc459-gdk6g_5fe485a1-e14f-4c09-b5b9-f252bc42b7e8/networking-console-plugin/0.log
2026-03-14 09:55:30.447310 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-source-55646444c4-trplf_9d751cbb-f2e2-430d-9754-c882a5e924a5/
2026-03-14 09:55:30.447327 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-source-55646444c4-trplf_9d751cbb-f2e2-430d-9754-c882a5e924a5/check-endpoints/
2026-03-14 09:55:30.447348 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-source-55646444c4-trplf_9d751cbb-f2e2-430d-9754-c882a5e924a5/check-endpoints/0.log
2026-03-14 09:55:30.447366 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-target-xd92c_3b6479f0-333b-4a96-9adf-2099afdc2447/
2026-03-14 09:55:30.447383 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-target-xd92c_3b6479f0-333b-4a96-9adf-2099afdc2447/network-check-target-container/
2026-03-14 09:55:30.447399 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-target-xd92c_3b6479f0-333b-4a96-9adf-2099afdc2447/network-check-target-container/0.log
2026-03-14 09:55:30.447416 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/
2026-03-14 09:55:30.447432 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/approver/
2026-03-14 09:55:30.447448 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/approver/0.log
2026-03-14 09:55:30.447464 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/webhook/
2026-03-14 09:55:30.447480 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/webhook/0.log
2026-03-14 09:55:30.447596 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_iptables-alerter-4ln5h_d75a4c96-2883-4a0b-bab2-0fab2b6c0b49/
2026-03-14 09:55:30.447618 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_iptables-alerter-4ln5h_d75a4c96-2883-4a0b-bab2-0fab2b6c0b49/iptables-alerter/
2026-03-14 09:55:30.447635 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_iptables-alerter-4ln5h_d75a4c96-2883-4a0b-bab2-0fab2b6c0b49/iptables-alerter/0.log
2026-03-14 09:55:30.447650 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_network-operator-58b4c7f79c-55gtf_37a5e44f-9a88-4405-be8a-b645485e7312/
2026-03-14 09:55:30.447666 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_network-operator-58b4c7f79c-55gtf_37a5e44f-9a88-4405-be8a-b645485e7312/network-operator/
2026-03-14 09:55:30.447682 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_network-operator-58b4c7f79c-55gtf_37a5e44f-9a88-4405-be8a-b645485e7312/network-operator/0.log
2026-03-14 09:55:30.447698 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-console-plugin-86f58fcf4-6w69h_986e9786-a2ab-4b67-8b7c-923951ef0928/
2026-03-14 09:55:30.447713 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-console-plugin-86f58fcf4-6w69h_986e9786-a2ab-4b67-8b7c-923951ef0928/nmstate-console-plugin/
2026-03-14 09:55:30.447729 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-console-plugin-86f58fcf4-6w69h_986e9786-a2ab-4b67-8b7c-923951ef0928/nmstate-console-plugin/0.log
2026-03-14 09:55:30.447744 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-handler-jz8xl_b0216839-cf58-4b58-8156-531abde2cbd1/
2026-03-14 09:55:30.447765 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-handler-jz8xl_b0216839-cf58-4b58-8156-531abde2cbd1/nmstate-handler/
2026-03-14 09:55:30.447781 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-handler-jz8xl_b0216839-cf58-4b58-8156-531abde2cbd1/nmstate-handler/0.log
2026-03-14 09:55:30.447901 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-9b8c8685d-wggcs_6c93f252-354d-484f-bc01-705766542dd9/
2026-03-14 09:55:30.447918 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-9b8c8685d-wggcs_6c93f252-354d-484f-bc01-705766542dd9/kube-rbac-proxy/
2026-03-14 09:55:30.447935 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-9b8c8685d-wggcs_6c93f252-354d-484f-bc01-705766542dd9/kube-rbac-proxy/0.log
2026-03-14 09:55:30.447951 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-9b8c8685d-wggcs_6c93f252-354d-484f-bc01-705766542dd9/nmstate-metrics/
2026-03-14 09:55:30.447967 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-9b8c8685d-wggcs_6c93f252-354d-484f-bc01-705766542dd9/nmstate-metrics/0.log
2026-03-14 09:55:30.447983 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-operator-796d4cfff4-scdwq_01314a74-350e-49d3-a090-a924ac031589/
2026-03-14 09:55:30.447999 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-operator-796d4cfff4-scdwq_01314a74-350e-49d3-a090-a924ac031589/nmstate-operator/
2026-03-14 09:55:30.448030 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-operator-796d4cfff4-scdwq_01314a74-350e-49d3-a090-a924ac031589/nmstate-operator/0.log
2026-03-14 09:55:30.448052 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-webhook-5f558f5558-wdzfb_e5ccd04c-58e5-49c2-9667-8cdaa37dd381/
2026-03-14 09:55:30.448069 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-webhook-5f558f5558-wdzfb_e5ccd04c-58e5-49c2-9667-8cdaa37dd381/nmstate-webhook/
2026-03-14 09:55:30.448178 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-webhook-5f558f5558-wdzfb_e5ccd04c-58e5-49c2-9667-8cdaa37dd381/nmstate-webhook/0.log
2026-03-14 09:55:30.448204 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-whkvt_ea621d77-5364-416c-a378-6adf0e89fc30/
2026-03-14 09:55:30.448221 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-whkvt_ea621d77-5364-416c-a378-6adf0e89fc30/fix-audit-permissions/
2026-03-14 09:55:30.448237 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-whkvt_ea621d77-5364-416c-a378-6adf0e89fc30/fix-audit-permissions/0.log
2026-03-14 09:55:30.448253 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-whkvt_ea621d77-5364-416c-a378-6adf0e89fc30/oauth-apiserver/
2026-03-14 09:55:30.448269 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-whkvt_ea621d77-5364-416c-a378-6adf0e89fc30/oauth-apiserver/0.log
2026-03-14 09:55:30.448285 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_catalog-operator-68c6474976-vj2ld_7ace8f29-86ea-40a3-8578-276187e6f1a2/
2026-03-14 09:55:30.448305 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_catalog-operator-68c6474976-vj2ld_7ace8f29-86ea-40a3-8578-276187e6f1a2/catalog-operator/
2026-03-14 09:55:30.448321 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_catalog-operator-68c6474976-vj2ld_7ace8f29-86ea-40a3-8578-276187e6f1a2/catalog-operator/0.log.gz
2026-03-14 09:55:30.448337 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29557995-d6pk8_e218bcb6-6204-4e1f-96a8-958d7f6460d5/
2026-03-14 09:55:30.448353 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29557995-d6pk8_e218bcb6-6204-4e1f-96a8-958d7f6460d5/collect-profiles/
2026-03-14 09:55:30.448369 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29557995-d6pk8_e218bcb6-6204-4e1f-96a8-958d7f6460d5/collect-profiles/0.log
2026-03-14 09:55:30.448384 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29558010-2fx5t_25604e27-1692-429b-bb26-e24e41514227/
2026-03-14 09:55:30.448400 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29558010-2fx5t_25604e27-1692-429b-bb26-e24e41514227/collect-profiles/
2026-03-14 09:55:30.448508 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29558010-2fx5t_25604e27-1692-429b-bb26-e24e41514227/collect-profiles/0.log
2026-03-14 09:55:30.448534 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29558025-lsgss_f7c84050-1abe-4cde-be35-a45eff902d68/
2026-03-14 09:55:30.448551 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29558025-lsgss_f7c84050-1abe-4cde-be35-a45eff902d68/collect-profiles/
2026-03-14 09:55:30.448567 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29558025-lsgss_f7c84050-1abe-4cde-be35-a45eff902d68/collect-profiles/0.log
2026-03-14 09:55:30.448583 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_olm-operator-6b444d44fb-jkv2j_581a4228-5ca8-400c-ba62-c16f4dddaddf/
2026-03-14 09:55:30.448599 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_olm-operator-6b444d44fb-jkv2j_581a4228-5ca8-400c-ba62-c16f4dddaddf/olm-operator/
2026-03-14 09:55:30.448615 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_olm-operator-6b444d44fb-jkv2j_581a4228-5ca8-400c-ba62-c16f4dddaddf/olm-operator/0.log
2026-03-14 09:55:30.448631 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-zc6w2_93b35280-bd38-4a0c-8192-807ce4f2eb0b/
2026-03-14 09:55:30.448646 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-zc6w2_93b35280-bd38-4a0c-8192-807ce4f2eb0b/kube-rbac-proxy/
2026-03-14 09:55:30.448662 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-zc6w2_93b35280-bd38-4a0c-8192-807ce4f2eb0b/kube-rbac-proxy/0.log
2026-03-14 09:55:30.448678 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-zc6w2_93b35280-bd38-4a0c-8192-807ce4f2eb0b/package-server-manager/
2026-03-14 09:55:30.448699 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-zc6w2_93b35280-bd38-4a0c-8192-807ce4f2eb0b/package-server-manager/0.log
2026-03-14 09:55:30.448819 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_packageserver-d55dfcdfc-b4kwx_e5ed96ac-e78f-424f-9aa0-f85a6f0321db/
2026-03-14 09:55:30.448837 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_packageserver-d55dfcdfc-b4kwx_e5ed96ac-e78f-424f-9aa0-f85a6f0321db/packageserver/
2026-03-14 09:55:30.448854 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_packageserver-d55dfcdfc-b4kwx_e5ed96ac-e78f-424f-9aa0-f85a6f0321db/packageserver/0.log
2026-03-14 09:55:30.448870 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-68bc856cb9-259ww_c9a78fe1-8b1a-4655-ad98-19c53622c2b1/
2026-03-14 09:55:30.448886 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-68bc856cb9-259ww_c9a78fe1-8b1a-4655-ad98-19c53622c2b1/prometheus-operator/
2026-03-14 09:55:30.448902 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-68bc856cb9-259ww_c9a78fe1-8b1a-4655-ad98-19c53622c2b1/prometheus-operator/0.log
2026-03-14 09:55:30.448918 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-6b9c87565f-llzfw_c888df81-af56-4ddd-8857-4952f199f288/
2026-03-14 09:55:30.448934 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-6b9c87565f-llzfw_c888df81-af56-4ddd-8857-4952f199f288/prometheus-operator-admission-webhook/
2026-03-14 09:55:30.448950 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-6b9c87565f-llzfw_c888df81-af56-4ddd-8857-4952f199f288/prometheus-operator-admission-webhook/0.log
2026-03-14 09:55:30.448966 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-6b9c87565f-wt77z_b9e5b9f8-a1eb-4cd9-8200-dbf2cbdb4193/
2026-03-14 09:55:30.448983 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-6b9c87565f-wt77z_b9e5b9f8-a1eb-4cd9-8200-dbf2cbdb4193/prometheus-operator-admission-webhook/
2026-03-14 09:55:30.448999 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-6b9c87565f-wt77z_b9e5b9f8-a1eb-4cd9-8200-dbf2cbdb4193/prometheus-operator-admission-webhook/0.log
2026-03-14 09:55:30.449028 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-operator-59bdc8b94-q6fzl_2dd43b3c-f9be-41a8-b1a6-11ab052283c7/
2026-03-14 09:55:30.449148 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-operator-59bdc8b94-q6fzl_2dd43b3c-f9be-41a8-b1a6-11ab052283c7/operator/
2026-03-14 09:55:30.449169 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-operator-59bdc8b94-q6fzl_2dd43b3c-f9be-41a8-b1a6-11ab052283c7/operator/0.log
2026-03-14 09:55:30.449186 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-ui-dashboards-66cbf594b5-hkwjk_0be4e37c-1708-4556-9ac0-e6daaf8fdadf/
2026-03-14 09:55:30.449202 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-ui-dashboards-66cbf594b5-hkwjk_0be4e37c-1708-4556-9ac0-e6daaf8fdadf/observability-ui-dashboards/
2026-03-14 09:55:30.449223 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-ui-dashboards-66cbf594b5-hkwjk_0be4e37c-1708-4556-9ac0-e6daaf8fdadf/observability-ui-dashboards/0.log
2026-03-14 09:55:30.449240 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_perses-operator-5bf474d74f-2vqmj_50a4be30-e002-4c15-b3ef-b3048665261b/
2026-03-14 09:55:30.449261 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_perses-operator-5bf474d74f-2vqmj_50a4be30-e002-4c15-b3ef-b3048665261b/perses-operator/
2026-03-14 09:55:30.449281 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_perses-operator-5bf474d74f-2vqmj_50a4be30-e002-4c15-b3ef-b3048665261b/perses-operator/0.log
2026-03-14 09:55:30.449301 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-mdp68_fba4fa81-8b34-4c43-aef8-84072b0bc8fb/
2026-03-14 09:55:30.449421 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-mdp68_fba4fa81-8b34-4c43-aef8-84072b0bc8fb/kube-rbac-proxy/
2026-03-14 09:55:30.449452 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-mdp68_fba4fa81-8b34-4c43-aef8-84072b0bc8fb/kube-rbac-proxy/0.log
2026-03-14 09:55:30.449474 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-mdp68_fba4fa81-8b34-4c43-aef8-84072b0bc8fb/ovnkube-cluster-manager/
2026-03-14 09:55:30.449495 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-mdp68_fba4fa81-8b34-4c43-aef8-84072b0bc8fb/ovnkube-cluster-manager/0.log
2026-03-14 09:55:30.449516 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qs67b_7ada84a5-3502-4dba-a59a-3742895ecf23/
2026-03-14 09:55:30.449536 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qs67b_7ada84a5-3502-4dba-a59a-3742895ecf23/kube-rbac-proxy-node/
2026-03-14 09:55:30.449650 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qs67b_7ada84a5-3502-4dba-a59a-3742895ecf23/kube-rbac-proxy-node/0.log
2026-03-14 09:55:30.449676 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qs67b_7ada84a5-3502-4dba-a59a-3742895ecf23/kube-rbac-proxy-ovn-metrics/
2026-03-14 09:55:30.449698 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qs67b_7ada84a5-3502-4dba-a59a-3742895ecf23/kube-rbac-proxy-ovn-metrics/0.log
2026-03-14 09:55:30.449718 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qs67b_7ada84a5-3502-4dba-a59a-3742895ecf23/kubecfg-setup/
2026-03-14 09:55:30.449823 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qs67b_7ada84a5-3502-4dba-a59a-3742895ecf23/kubecfg-setup/0.log
2026-03-14 09:55:30.449855 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qs67b_7ada84a5-3502-4dba-a59a-3742895ecf23/nbdb/
2026-03-14 09:55:30.449877 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qs67b_7ada84a5-3502-4dba-a59a-3742895ecf23/nbdb/0.log
2026-03-14 09:55:30.449897 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qs67b_7ada84a5-3502-4dba-a59a-3742895ecf23/northd/
2026-03-14 09:55:30.449918 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qs67b_7ada84a5-3502-4dba-a59a-3742895ecf23/northd/0.log
2026-03-14 09:55:30.450009 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qs67b_7ada84a5-3502-4dba-a59a-3742895ecf23/ovn-acl-logging/
2026-03-14 09:55:30.450154 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qs67b_7ada84a5-3502-4dba-a59a-3742895ecf23/ovn-acl-logging/0.log
2026-03-14 09:55:30.450176 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qs67b_7ada84a5-3502-4dba-a59a-3742895ecf23/ovn-controller/
2026-03-14 09:55:30.450192 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qs67b_7ada84a5-3502-4dba-a59a-3742895ecf23/ovn-controller/0.log
2026-03-14 09:55:30.450208 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qs67b_7ada84a5-3502-4dba-a59a-3742895ecf23/ovnkube-controller/
2026-03-14 09:55:30.450224 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qs67b_7ada84a5-3502-4dba-a59a-3742895ecf23/ovnkube-controller/0.log.gz
2026-03-14 09:55:30.450239 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qs67b_7ada84a5-3502-4dba-a59a-3742895ecf23/sbdb/
2026-03-14 09:55:30.450254 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-qs67b_7ada84a5-3502-4dba-a59a-3742895ecf23/sbdb/0.log
2026-03-14 09:55:30.450269 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-route-controller-manager_route-controller-manager-dfb87fb89-5x8mg_a2c55e27-5f7d-42f8-b2a8-4eac60c4c7be/
2026-03-14 09:55:30.450285 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-route-controller-manager_route-controller-manager-dfb87fb89-5x8mg_a2c55e27-5f7d-42f8-b2a8-4eac60c4c7be/route-controller-manager/
2026-03-14 09:55:30.450300 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-route-controller-manager_route-controller-manager-dfb87fb89-5x8mg_a2c55e27-5f7d-42f8-b2a8-4eac60c4c7be/route-controller-manager/0.log
2026-03-14 09:55:30.450316 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca-operator_service-ca-operator-777779d784-zd8f5_b827bdf7-bbd1-4b57-8485-09a312e20b43/
2026-03-14 09:55:30.450431 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca-operator_service-ca-operator-777779d784-zd8f5_b827bdf7-bbd1-4b57-8485-09a312e20b43/service-ca-operator/
2026-03-14 09:55:30.450456 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca-operator_service-ca-operator-777779d784-zd8f5_b827bdf7-bbd1-4b57-8485-09a312e20b43/service-ca-operator/0.log
2026-03-14 09:55:30.450474 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca_service-ca-9c57cc56f-dfkjs_b315e659-0690-463c-88c4-659124922ddc/
2026-03-14 09:55:30.450490 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca_service-ca-9c57cc56f-dfkjs_b315e659-0690-463c-88c4-659124922ddc/service-ca-controller/
2026-03-14 09:55:30.450506 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca_service-ca-9c57cc56f-dfkjs_b315e659-0690-463c-88c4-659124922ddc/service-ca-controller/0.log
2026-03-14 09:55:30.450611 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_62d0bed58019ccc1d3626e895c0dbede7faaf3676e025b9f97c0e0a616vmnnb_aec90a2d-58f3-4dee-90c3-60a8fc90bc56/
2026-03-14 09:55:30.450637 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_62d0bed58019ccc1d3626e895c0dbede7faaf3676e025b9f97c0e0a616vmnnb_aec90a2d-58f3-4dee-90c3-60a8fc90bc56/extract/
2026-03-14 09:55:30.450659 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_62d0bed58019ccc1d3626e895c0dbede7faaf3676e025b9f97c0e0a616vmnnb_aec90a2d-58f3-4dee-90c3-60a8fc90bc56/extract/0.log
2026-03-14 09:55:30.450676 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_62d0bed58019ccc1d3626e895c0dbede7faaf3676e025b9f97c0e0a616vmnnb_aec90a2d-58f3-4dee-90c3-60a8fc90bc56/pull/
2026-03-14 09:55:30.450694 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_62d0bed58019ccc1d3626e895c0dbede7faaf3676e025b9f97c0e0a616vmnnb_aec90a2d-58f3-4dee-90c3-60a8fc90bc56/pull/0.log
2026-03-14 09:55:30.450710 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_62d0bed58019ccc1d3626e895c0dbede7faaf3676e025b9f97c0e0a616vmnnb_aec90a2d-58f3-4dee-90c3-60a8fc90bc56/util/
2026-03-14 09:55:30.450814 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_62d0bed58019ccc1d3626e895c0dbede7faaf3676e025b9f97c0e0a616vmnnb_aec90a2d-58f3-4dee-90c3-60a8fc90bc56/util/0.log
2026-03-14 09:55:30.450844 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_632ec92be58b5de1327ed8930598ec28f26f424a3aad5e3b3c0e97c82cpbhtr_07597b31-3cd5-4c5c-8bb1-65366038ddbb/
2026-03-14 09:55:30.450862 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_632ec92be58b5de1327ed8930598ec28f26f424a3aad5e3b3c0e97c82cpbhtr_07597b31-3cd5-4c5c-8bb1-65366038ddbb/extract/
2026-03-14 09:55:30.450880 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_632ec92be58b5de1327ed8930598ec28f26f424a3aad5e3b3c0e97c82cpbhtr_07597b31-3cd5-4c5c-8bb1-65366038ddbb/extract/0.log
2026-03-14 09:55:30.450896 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_632ec92be58b5de1327ed8930598ec28f26f424a3aad5e3b3c0e97c82cpbhtr_07597b31-3cd5-4c5c-8bb1-65366038ddbb/pull/
2026-03-14 09:55:30.451043 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_632ec92be58b5de1327ed8930598ec28f26f424a3aad5e3b3c0e97c82cpbhtr_07597b31-3cd5-4c5c-8bb1-65366038ddbb/pull/0.log
2026-03-14 09:55:30.451080 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_632ec92be58b5de1327ed8930598ec28f26f424a3aad5e3b3c0e97c82cpbhtr_07597b31-3cd5-4c5c-8bb1-65366038ddbb/util/
2026-03-14 09:55:30.451109 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_632ec92be58b5de1327ed8930598ec28f26f424a3aad5e3b3c0e97c82cpbhtr_07597b31-3cd5-4c5c-8bb1-65366038ddbb/util/0.log
2026-03-14 09:55:30.451126 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_barbican-operator-controller-manager-64768694d-m9dwm_b3749fc9-e22e-42b9-8865-68679f7d78f1/
2026-03-14 09:55:30.451142 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_barbican-operator-controller-manager-64768694d-m9dwm_b3749fc9-e22e-42b9-8865-68679f7d78f1/manager/
2026-03-14 09:55:30.451158 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_barbican-operator-controller-manager-64768694d-m9dwm_b3749fc9-e22e-42b9-8865-68679f7d78f1/manager/0.log.gz
2026-03-14 09:55:30.451174 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_cinder-operator-controller-manager-cb6d66846-b5jzw_9bf45de7-ba46-4ce9-a7d7-fc26e253423b/
2026-03-14 09:55:30.451190 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_cinder-operator-controller-manager-cb6d66846-b5jzw_9bf45de7-ba46-4ce9-a7d7-fc26e253423b/manager/
2026-03-14 09:55:30.451210 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_cinder-operator-controller-manager-cb6d66846-b5jzw_9bf45de7-ba46-4ce9-a7d7-fc26e253423b/manager/0.log.gz
2026-03-14 09:55:30.451227 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_designate-operator-controller-manager-9c8c85cd7-8xwbp_6d1aed1b-6436-46ca-a824-59eafb8ca5d3/
2026-03-14 09:55:30.451243 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_designate-operator-controller-manager-9c8c85cd7-8xwbp_6d1aed1b-6436-46ca-a824-59eafb8ca5d3/manager/
2026-03-14 09:55:30.451367 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_designate-operator-controller-manager-9c8c85cd7-8xwbp_6d1aed1b-6436-46ca-a824-59eafb8ca5d3/manager/0.log
2026-03-14 09:55:30.451390 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_glance-operator-controller-manager-74d565fbd5-c924b_18021394-f27d-422e-a68c-24a19d74ceb8/
2026-03-14 09:55:30.451407 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_glance-operator-controller-manager-74d565fbd5-c924b_18021394-f27d-422e-a68c-24a19d74ceb8/manager/
2026-03-14 09:55:30.451423 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_glance-operator-controller-manager-74d565fbd5-c924b_18021394-f27d-422e-a68c-24a19d74ceb8/manager/0.log
2026-03-14 09:55:30.451439 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_heat-operator-controller-manager-6d6bd468b-db2v8_876c14aa-a86a-495f-a110-3ade7d8d69fb/
2026-03-14 09:55:30.451455 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_heat-operator-controller-manager-6d6bd468b-db2v8_876c14aa-a86a-495f-a110-3ade7d8d69fb/manager/
2026-03-14 09:55:30.451471 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_heat-operator-controller-manager-6d6bd468b-db2v8_876c14aa-a86a-495f-a110-3ade7d8d69fb/manager/0.log
2026-03-14 09:55:30.451487 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_horizon-operator-controller-manager-5b9475cdd7-586s5_4492ee98-efe4-49c3-8c14-86453a8e8714/
2026-03-14 09:55:30.451502 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_horizon-operator-controller-manager-5b9475cdd7-586s5_4492ee98-efe4-49c3-8c14-86453a8e8714/manager/
2026-03-14 09:55:30.451518 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_horizon-operator-controller-manager-5b9475cdd7-586s5_4492ee98-efe4-49c3-8c14-86453a8e8714/manager/0.log
2026-03-14 09:55:30.451533 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_infra-operator-controller-manager-fbfb5bd65-v7cqm_e43b6f19-e463-43ce-9efe-5cefa3b53682/
2026-03-14 09:55:30.451549 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_infra-operator-controller-manager-fbfb5bd65-v7cqm_e43b6f19-e463-43ce-9efe-5cefa3b53682/manager/
2026-03-14 09:55:30.451564 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_infra-operator-controller-manager-fbfb5bd65-v7cqm_e43b6f19-e463-43ce-9efe-5cefa3b53682/manager/0.log
2026-03-14 09:55:30.451579 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ironic-operator-controller-manager-bf6b7fd8c-56k2s_401a3c6d-db2a-435b-b7f5-08816736d895/
2026-03-14 09:55:30.451723 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ironic-operator-controller-manager-bf6b7fd8c-56k2s_401a3c6d-db2a-435b-b7f5-08816736d895/manager/
2026-03-14 09:55:30.451742 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ironic-operator-controller-manager-bf6b7fd8c-56k2s_401a3c6d-db2a-435b-b7f5-08816736d895/manager/0.log
2026-03-14 09:55:30.451762 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_keystone-operator-controller-manager-68f8d496f8-4knlv_18ad86e0-d070-4de1-bd50-a93f9abdf715/
2026-03-14 09:55:30.451779 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_keystone-operator-controller-manager-68f8d496f8-4knlv_18ad86e0-d070-4de1-bd50-a93f9abdf715/manager/
2026-03-14 09:55:30.451795 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_keystone-operator-controller-manager-68f8d496f8-4knlv_18ad86e0-d070-4de1-bd50-a93f9abdf715/manager/0.log
2026-03-14 09:55:30.451810 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_manila-operator-controller-manager-6f6f57b9b6-9c7lm_19ee7ede-7bda-46bc-8413-95262fa53969/
2026-03-14 09:55:30.451826 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_manila-operator-controller-manager-6f6f57b9b6-9c7lm_19ee7ede-7bda-46bc-8413-95262fa53969/manager/
2026-03-14 09:55:30.451841 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_manila-operator-controller-manager-6f6f57b9b6-9c7lm_19ee7ede-7bda-46bc-8413-95262fa53969/manager/0.log
2026-03-14 09:55:30.451856 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_mariadb-operator-controller-manager-744456f686-nnfnk_03bd3900-f5fa-476e-a91a-f492e4a424dc/
2026-03-14 09:55:30.451871 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_mariadb-operator-controller-manager-744456f686-nnfnk_03bd3900-f5fa-476e-a91a-f492e4a424dc/manager/
2026-03-14 09:55:30.451887 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_mariadb-operator-controller-manager-744456f686-nnfnk_03bd3900-f5fa-476e-a91a-f492e4a424dc/manager/0.log
2026-03-14 09:55:30.451902 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_neutron-operator-controller-manager-645c9f6488-9bn7h_240f1b23-5499-4644-a442-9647e71a33d4/
2026-03-14 09:55:30.451917 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_neutron-operator-controller-manager-645c9f6488-9bn7h_240f1b23-5499-4644-a442-9647e71a33d4/manager/
2026-03-14 09:55:30.451932 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_neutron-operator-controller-manager-645c9f6488-9bn7h_240f1b23-5499-4644-a442-9647e71a33d4/manager/0.log
2026-03-14 09:55:30.451947 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_nova-operator-controller-manager-58ff56fcc7-9kjv4_fdc31f77-84df-4657-b5ec-a7fcd8b673e2/
2026-03-14 09:55:30.451962 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_nova-operator-controller-manager-58ff56fcc7-9kjv4_fdc31f77-84df-4657-b5ec-a7fcd8b673e2/manager/
2026-03-14 09:55:30.452131 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_nova-operator-controller-manager-58ff56fcc7-9kjv4_fdc31f77-84df-4657-b5ec-a7fcd8b673e2/manager/0.log
2026-03-14 09:55:30.452166 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_octavia-operator-controller-manager-7cf9f49d6-qwdtd_e0b8fc0f-4ffa-4c26-84e4-9613f5161286/
2026-03-14 09:55:30.452185 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_octavia-operator-controller-manager-7cf9f49d6-qwdtd_e0b8fc0f-4ffa-4c26-84e4-9613f5161286/manager/
2026-03-14 09:55:30.452202 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_octavia-operator-controller-manager-7cf9f49d6-qwdtd_e0b8fc0f-4ffa-4c26-84e4-9613f5161286/manager/0.log
2026-03-14 09:55:30.452223 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-baremetal-operator-controller-manager-c5677dc5d-xbdph_67182d33-3abc-4661-8614-94238efc9e45/
2026-03-14 09:55:30.452239 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-baremetal-operator-controller-manager-c5677dc5d-xbdph_67182d33-3abc-4661-8614-94238efc9e45/manager/
2026-03-14 09:55:30.452254 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-baremetal-operator-controller-manager-c5677dc5d-xbdph_67182d33-3abc-4661-8614-94238efc9e45/manager/0.log
2026-03-14 09:55:30.452269 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-controller-manager-59b5586c67-c9k5q_6bff6422-f245-457e-9ddb-28f957c9edac/
2026-03-14 09:55:30.452285 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-controller-manager-59b5586c67-c9k5q_6bff6422-f245-457e-9ddb-28f957c9edac/manager/
2026-03-14 09:55:30.452300 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-controller-manager-59b5586c67-c9k5q_6bff6422-f245-457e-9ddb-28f957c9edac/manager/0.log.gz
2026-03-14 09:55:30.452316 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-index-kgjkt_9ac17263-727b-4bdd-8217-920844d59367/
2026-03-14 09:55:30.452331 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-index-kgjkt_9ac17263-727b-4bdd-8217-920844d59367/registry-server/
2026-03-14 09:55:30.452346 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-index-kgjkt_9ac17263-727b-4bdd-8217-920844d59367/registry-server/0.log
2026-03-14 09:55:30.452361 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ovn-operator-controller-manager-848d74f969-6n97h_a8d7db47-dd9b-4785-8e80-d7d97a324225/
2026-03-14 09:55:30.452376 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ovn-operator-controller-manager-848d74f969-6n97h_a8d7db47-dd9b-4785-8e80-d7d97a324225/manager/
2026-03-14 09:55:30.452391 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ovn-operator-controller-manager-848d74f969-6n97h_a8d7db47-dd9b-4785-8e80-d7d97a324225/manager/0.log
2026-03-14 09:55:30.452406 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_placement-operator-controller-manager-b5c469fd-bffxq_bad25fda-3055-4fa2-8fd4-24980a88c7c6/
2026-03-14 09:55:30.452421 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_placement-operator-controller-manager-b5c469fd-bffxq_bad25fda-3055-4fa2-8fd4-24980a88c7c6/manager/
2026-03-14 09:55:30.452532 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_placement-operator-controller-manager-b5c469fd-bffxq_bad25fda-3055-4fa2-8fd4-24980a88c7c6/manager/0.log
2026-03-14 09:55:30.452557 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_rabbitmq-cluster-operator-manager-668c99d594-vppb7_3d61bf91-4992-47ad-8a53-e823a71d3f9c/
2026-03-14 09:55:30.452575 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_rabbitmq-cluster-operator-manager-668c99d594-vppb7_3d61bf91-4992-47ad-8a53-e823a71d3f9c/operator/
2026-03-14 09:55:30.452591 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_rabbitmq-cluster-operator-manager-668c99d594-vppb7_3d61bf91-4992-47ad-8a53-e823a71d3f9c/operator/0.log
2026-03-14 09:55:30.452607 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_swift-operator-controller-manager-7f7469dbc6-6djhw_24e7f369-d268-4aa7-89f9-1b4ce48fd197/
2026-03-14 09:55:30.452628 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_swift-operator-controller-manager-7f7469dbc6-6djhw_24e7f369-d268-4aa7-89f9-1b4ce48fd197/manager/
2026-03-14 09:55:30.452645 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_swift-operator-controller-manager-7f7469dbc6-6djhw_24e7f369-d268-4aa7-89f9-1b4ce48fd197/manager/0.log
2026-03-14 09:55:30.452660 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_telemetry-operator-controller-manager-6646df7cdb-tspwx_2c1f44c6-aae3-4c3c-933e-c87956fb0fe6/
2026-03-14 09:55:30.452676 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_telemetry-operator-controller-manager-6646df7cdb-tspwx_2c1f44c6-aae3-4c3c-933e-c87956fb0fe6/manager/
2026-03-14 09:55:30.452879 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_telemetry-operator-controller-manager-6646df7cdb-tspwx_2c1f44c6-aae3-4c3c-933e-c87956fb0fe6/manager/0.log.gz
2026-03-14 09:55:30.452902 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_test-operator-controller-manager-8467ccb4c8-r9dtz_f1b524ae-60fa-48a7-aa07-bf354bd2ff62/
2026-03-14 09:55:30.452919 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_test-operator-controller-manager-8467ccb4c8-r9dtz_f1b524ae-60fa-48a7-aa07-bf354bd2ff62/manager/
2026-03-14 09:55:30.452936 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_test-operator-controller-manager-8467ccb4c8-r9dtz_f1b524ae-60fa-48a7-aa07-bf354bd2ff62/manager/0.log
2026-03-14 09:55:30.452952 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-controller-manager-788cc4b948-xgdnd_75ec3cea-39db-4cc8-8065-17259b7dd1e4/
2026-03-14 09:55:30.452969 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-controller-manager-788cc4b948-xgdnd_75ec3cea-39db-4cc8-8065-17259b7dd1e4/manager/
2026-03-14 09:55:30.452985 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-controller-manager-788cc4b948-xgdnd_75ec3cea-39db-4cc8-8065-17259b7dd1e4/manager/0.log.gz
2026-03-14 09:55:30.453002 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-index-jl5p2_75e6f1a9-bb4c-496d-841c-849bf8a375d7/
2026-03-14 09:55:30.453049 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-index-jl5p2_75e6f1a9-bb4c-496d-841c-849bf8a375d7/registry-server/
2026-03-14 09:55:30.453070 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-index-jl5p2_75e6f1a9-bb4c-496d-841c-849bf8a375d7/registry-server/0.log
2026-03-14 09:55:30.453678 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_f4bd2bca-f8c8-4cf0-a8b3-0bc6e5951531/
2026-03-14 09:55:30.453724 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_f4bd2bca-f8c8-4cf0-a8b3-0bc6e5951531/alertmanager/
2026-03-14 09:55:30.453744 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_f4bd2bca-f8c8-4cf0-a8b3-0bc6e5951531/alertmanager/0.log
2026-03-14 09:55:30.453761 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_f4bd2bca-f8c8-4cf0-a8b3-0bc6e5951531/config-reloader/
2026-03-14 09:55:30.453777 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_f4bd2bca-f8c8-4cf0-a8b3-0bc6e5951531/config-reloader/0.log
2026-03-14 09:55:30.453799 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_f4bd2bca-f8c8-4cf0-a8b3-0bc6e5951531/init-config-reloader/
2026-03-14 09:55:30.453820 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_f4bd2bca-f8c8-4cf0-a8b3-0bc6e5951531/init-config-reloader/0.log
2026-03-14 09:55:30.453838 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_3f4eb2d1-4293-4a16-984e-f3aaae12b369/
2026-03-14 09:55:30.453854 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_3f4eb2d1-4293-4a16-984e-f3aaae12b369/ceilometer-central-agent/
2026-03-14 09:55:30.454006 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_3f4eb2d1-4293-4a16-984e-f3aaae12b369/ceilometer-central-agent/0.log
2026-03-14 09:55:30.454057 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_3f4eb2d1-4293-4a16-984e-f3aaae12b369/ceilometer-notification-agent/
2026-03-14 09:55:30.454075 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_3f4eb2d1-4293-4a16-984e-f3aaae12b369/ceilometer-notification-agent/0.log
2026-03-14 09:55:30.454092 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_3f4eb2d1-4293-4a16-984e-f3aaae12b369/proxy-httpd/
2026-03-14 09:55:30.454107 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_3f4eb2d1-4293-4a16-984e-f3aaae12b369/proxy-httpd/0.log
2026-03-14 09:55:30.454123 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_3f4eb2d1-4293-4a16-984e-f3aaae12b369/sg-core/
2026-03-14 09:55:30.454138 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_3f4eb2d1-4293-4a16-984e-f3aaae12b369/sg-core/0.log
2026-03-14 09:55:30.454153 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-75c5988f99-28ht4_67a35faf-96ec-45eb-af07-2a248bca71e2/
2026-03-14 09:55:30.454169 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-75c5988f99-28ht4_67a35faf-96ec-45eb-af07-2a248bca71e2/keystone-api/
2026-03-14 09:55:30.454184 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-75c5988f99-28ht4_67a35faf-96ec-45eb-af07-2a248bca71e2/keystone-api/0.log
2026-03-14 09:55:30.454199 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_kube-state-metrics-0_0e00a078-7836-4348-b513-0c8af77d837d/
2026-03-14 09:55:30.454214 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_kube-state-metrics-0_0e00a078-7836-4348-b513-0c8af77d837d/kube-state-metrics/
2026-03-14 09:55:30.454262 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_kube-state-metrics-0_0e00a078-7836-4348-b513-0c8af77d837d/kube-state-metrics/0.log
2026-03-14 09:55:30.454384 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_memcached-0_2973e91c-c0d8-4a9c-871e-0147ecc86297/
2026-03-14 09:55:30.454409 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_memcached-0_2973e91c-c0d8-4a9c-871e-0147ecc86297/memcached/
2026-03-14 09:55:30.454426 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_memcached-0_2973e91c-c0d8-4a9c-871e-0147ecc86297/memcached/0.log.gz
2026-03-14 09:55:30.454446 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_19027c02-7e5c-484e-ba9f-b9f9e7c4f81e/
2026-03-14 09:55:30.454463 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_19027c02-7e5c-484e-ba9f-b9f9e7c4f81e/galera/
2026-03-14 09:55:30.454478 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_19027c02-7e5c-484e-ba9f-b9f9e7c4f81e/galera/0.log
2026-03-14 09:55:30.454493 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_19027c02-7e5c-484e-ba9f-b9f9e7c4f81e/mysql-bootstrap/
2026-03-14 09:55:30.454509 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_19027c02-7e5c-484e-ba9f-b9f9e7c4f81e/mysql-bootstrap/0.log
2026-03-14 09:55:30.454524 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstackclient_4fbb48a6-d5ed-423b-90e4-809c839c8675/
2026-03-14 09:55:30.454539 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstackclient_4fbb48a6-d5ed-423b-90e4-809c839c8675/openstackclient/
2026-03-14 09:55:30.454555 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstackclient_4fbb48a6-d5ed-423b-90e4-809c839c8675/openstackclient/0.log
2026-03-14 09:55:30.454570 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_51fad570-b371-4486-9a35-bea145391f8f/
2026-03-14 09:55:30.454690 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_51fad570-b371-4486-9a35-bea145391f8f/config-reloader/
2026-03-14 09:55:30.454715 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_51fad570-b371-4486-9a35-bea145391f8f/config-reloader/0.log
2026-03-14 09:55:30.454731 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_51fad570-b371-4486-9a35-bea145391f8f/init-config-reloader/
2026-03-14 09:55:30.454747 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_51fad570-b371-4486-9a35-bea145391f8f/init-config-reloader/0.log
2026-03-14 09:55:30.454763 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_51fad570-b371-4486-9a35-bea145391f8f/prometheus/
2026-03-14 09:55:30.454778 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_51fad570-b371-4486-9a35-bea145391f8f/prometheus/0.log
2026-03-14 09:55:30.454794 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_51fad570-b371-4486-9a35-bea145391f8f/thanos-sidecar/
2026-03-14 09:55:30.454809 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_51fad570-b371-4486-9a35-bea145391f8f/thanos-sidecar/0.log
2026-03-14 09:55:30.454824 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_ca474341-5bf4-4fa5-bc46-56d42c3ccffd/
2026-03-14 09:55:30.454839 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_ca474341-5bf4-4fa5-bc46-56d42c3ccffd/rabbitmq/
2026-03-14 09:55:30.454854 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_ca474341-5bf4-4fa5-bc46-56d42c3ccffd/rabbitmq/0.log
2026-03-14 09:55:30.454874 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_ca474341-5bf4-4fa5-bc46-56d42c3ccffd/setup-container/
2026-03-14 09:55:30.454890 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_ca474341-5bf4-4fa5-bc46-56d42c3ccffd/setup-container/0.log
2026-03-14 09:55:30.455137 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_28cef006-3a3a-464f-b6d0-9faea75b0a9e/
2026-03-14 09:55:30.455161 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_28cef006-3a3a-464f-b6d0-9faea75b0a9e/rabbitmq/
2026-03-14 09:55:30.455178 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_28cef006-3a3a-464f-b6d0-9faea75b0a9e/rabbitmq/0.log
2026-03-14 09:55:30.455194 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_28cef006-3a3a-464f-b6d0-9faea75b0a9e/setup-container/
2026-03-14 09:55:30.455211 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_28cef006-3a3a-464f-b6d0-9faea75b0a9e/setup-container/0.log
2026-03-14 09:55:30.455228 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/
2026-03-14 09:55:30.455245 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/event-filter.html
2026-03-14 09:55:30.455262 | controller | cL+++++++++ ci-framework-data/logs/openstack-must-gather/latest -> quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3
2026-03-14 09:55:30.455279 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/must-gather.logs
2026-03-14 09:55:30.455295 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/timestamp
2026-03-14 09:55:30.455312 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/
2026-03-14 09:55:30.455328 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/all-containers-cpu-top.log
2026-03-14 09:55:30.455344 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/gather.logs
2026-03-14 09:55:30.455525 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/must-gather.tar.xz
2026-03-14 09:55:30.455549 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/packagemanifests.gz
2026-03-14 09:55:30.455566 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/run.log
2026-03-14 09:55:30.455583 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/version
2026-03-14 09:55:30.455600 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/
2026-03-14 09:55:30.455616 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.barbican.openstack.org.yaml
2026-03-14 09:55:30.455806 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.baremetal.openstack.org.yaml
2026-03-14 09:55:30.455836 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.cinder.openstack.org.yaml
2026-03-14 09:55:30.455854 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.client.openstack.org.yaml
2026-03-14 09:55:30.455871 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.core.openstack.org.yaml
2026-03-14 09:55:30.455887 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.dataplane.openstack.org.yaml
2026-03-14 09:55:30.456062 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.designate.openstack.org.yaml
2026-03-14 09:55:30.456089 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.glance.openstack.org.yaml
2026-03-14 09:55:30.456106 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.heat.openstack.org.yaml
2026-03-14 09:55:30.456122 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.horizon.openstack.org.yaml
2026-03-14 09:55:30.456138 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.instanceha.openstack.org.yaml
2026-03-14 09:55:30.456153 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.ironic.openstack.org.yaml
2026-03-14 09:55:30.456169 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.keystone.openstack.org.yaml
2026-03-14 09:55:30.456184 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.manila.openstack.org.yaml
2026-03-14 09:55:30.456200 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.mariadb.openstack.org.yaml
2026-03-14 09:55:30.456215 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.memcached.openstack.org.yaml
2026-03-14 09:55:30.456236 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.network.openstack.org.yaml
2026-03-14 09:55:30.456253 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.neutron.openstack.org.yaml
2026-03-14 09:55:30.456269 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.nova.openstack.org.yaml
2026-03-14 09:55:30.456285 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.octavia.openstack.org.yaml
2026-03-14 09:55:30.456301 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.operator.openstack.org.yaml
2026-03-14 09:55:30.456422 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.ovn.openstack.org.yaml
2026-03-14 09:55:30.456441 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.placement.openstack.org.yaml
2026-03-14 09:55:30.456457 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.rabbitmq.com.yaml
2026-03-14 09:55:30.456473 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.rabbitmq.openstack.org.yaml
2026-03-14 09:55:30.456489 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.redis.openstack.org.yaml
2026-03-14 09:55:30.456505 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.swift.openstack.org.yaml
2026-03-14 09:55:30.456521 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.telemetry.openstack.org.yaml
2026-03-14 09:55:30.456537 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.test.openstack.org.yaml
2026-03-14 09:55:30.456553 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.topology.openstack.org.yaml
2026-03-14 09:55:30.456569 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/apiservices/v1beta1.watcher.openstack.org.yaml
2026-03-14 09:55:30.456589 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/
2026-03-14 09:55:30.456708 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ansibletests.test.openstack.org.yaml
2026-03-14 09:55:30.456733 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/autoscalings.telemetry.openstack.org.yaml
2026-03-14 09:55:30.456751 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/barbicanapis.barbican.openstack.org.yaml
2026-03-14 09:55:30.456767 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/barbicankeystonelisteners.barbican.openstack.org.yaml
2026-03-14 09:55:30.456783 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/barbicans.barbican.openstack.org.yaml
2026-03-14 09:55:30.456798 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/barbicanworkers.barbican.openstack.org.yaml
2026-03-14 09:55:30.456961 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/bgpconfigurations.network.openstack.org.yaml
2026-03-14 09:55:30.456987 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ceilometers.telemetry.openstack.org.yaml
2026-03-14 09:55:30.457005 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cinderapis.cinder.openstack.org.yaml
2026-03-14 09:55:30.457051 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cinderbackups.cinder.openstack.org.yaml
2026-03-14 09:55:30.457071 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cinders.cinder.openstack.org.yaml
2026-03-14 09:55:30.457088 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cinderschedulers.cinder.openstack.org.yaml
2026-03-14 09:55:30.457104 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cindervolumes.cinder.openstack.org.yaml
2026-03-14 09:55:30.457120 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cloudkitties.telemetry.openstack.org.yaml
2026-03-14 09:55:30.457209 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cloudkittyapis.telemetry.openstack.org.yaml
2026-03-14 09:55:30.457237 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/cloudkittyprocs.telemetry.openstack.org.yaml
2026-03-14 09:55:30.457254 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designateapis.designate.openstack.org.yaml
2026-03-14 09:55:30.457270 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designatebackendbind9s.designate.openstack.org.yaml
2026-03-14 09:55:30.457286 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designatecentrals.designate.openstack.org.yaml
2026-03-14 09:55:30.457302 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designatemdnses.designate.openstack.org.yaml
2026-03-14 09:55:30.457318 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designateproducers.designate.openstack.org.yaml
2026-03-14 09:55:30.457334 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designates.designate.openstack.org.yaml
2026-03-14 09:55:30.457351 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designateunbounds.designate.openstack.org.yaml
2026-03-14 09:55:30.457367 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/designateworkers.designate.openstack.org.yaml
2026-03-14 09:55:30.457383 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/dnsdata.network.openstack.org.yaml
2026-03-14 09:55:30.457398 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/dnsmasqs.network.openstack.org.yaml
2026-03-14 09:55:30.457414 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/galerabackups.mariadb.openstack.org.yaml
2026-03-14 09:55:30.457430 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/galerarestores.mariadb.openstack.org.yaml
2026-03-14 09:55:30.457445 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/galeras.mariadb.openstack.org.yaml
2026-03-14 09:55:30.457465 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/glanceapis.glance.openstack.org.yaml
2026-03-14 09:55:30.457481 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/glances.glance.openstack.org.yaml
2026-03-14 09:55:30.457575 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/heatapis.heat.openstack.org.yaml
2026-03-14 09:55:30.457596 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/heatcfnapis.heat.openstack.org.yaml
2026-03-14 09:55:30.457612 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/heatengines.heat.openstack.org.yaml
2026-03-14 09:55:30.457629 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/heats.heat.openstack.org.yaml
2026-03-14 09:55:30.457645 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/horizons.horizon.openstack.org.yaml
2026-03-14 09:55:30.457661 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/horizontests.test.openstack.org.yaml
2026-03-14 09:55:30.457677 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/instancehas.instanceha.openstack.org.yaml
2026-03-14 09:55:30.457692 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ipsets.network.openstack.org.yaml
2026-03-14 09:55:30.457709 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ironicapis.ironic.openstack.org.yaml
2026-03-14 09:55:30.457725 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ironicconductors.ironic.openstack.org.yaml
2026-03-14 09:55:30.457741 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ironicinspectors.ironic.openstack.org.yaml
2026-03-14 09:55:30.457832 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ironicneutronagents.ironic.openstack.org.yaml
2026-03-14 09:55:30.457863 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ironics.ironic.openstack.org.yaml
2026-03-14 09:55:30.457881 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/keystoneapis.keystone.openstack.org.yaml
2026-03-14 09:55:30.457902 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/keystoneapplicationcredentials.keystone.openstack.org.yaml
2026-03-14 09:55:30.457918 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/keystoneendpoints.keystone.openstack.org.yaml
2026-03-14 09:55:30.457934 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/keystoneservices.keystone.openstack.org.yaml
2026-03-14 09:55:30.457949 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/loggings.telemetry.openstack.org.yaml
2026-03-14 09:55:30.457965 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/manilaapis.manila.openstack.org.yaml
2026-03-14 09:55:30.457980 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/manilas.manila.openstack.org.yaml
2026-03-14 09:55:30.457996 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/manilaschedulers.manila.openstack.org.yaml
2026-03-14 09:55:30.458011 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/manilashares.manila.openstack.org.yaml
2026-03-14 09:55:30.458059 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/mariadbaccounts.mariadb.openstack.org.yaml
2026-03-14 09:55:30.458076 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/mariadbdatabases.mariadb.openstack.org.yaml
2026-03-14 09:55:30.458092 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/memcacheds.memcached.openstack.org.yaml
2026-03-14 09:55:30.458107 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/metricstorages.telemetry.openstack.org.yaml
2026-03-14 09:55:30.458122 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/netconfigs.network.openstack.org.yaml
2026-03-14 09:55:30.458137 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/neutronapis.neutron.openstack.org.yaml
2026-03-14 09:55:30.458152 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/nova.nova.openstack.org.yaml
2026-03-14 09:55:30.458172 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novaapis.nova.openstack.org.yaml
2026-03-14 09:55:30.458188 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novacells.nova.openstack.org.yaml
2026-03-14 09:55:30.458763 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novacomputes.nova.openstack.org.yaml
2026-03-14 09:55:30.458784 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novaconductors.nova.openstack.org.yaml
2026-03-14 09:55:30.458801 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novametadata.nova.openstack.org.yaml
2026-03-14 09:55:30.458817 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novanovncproxies.nova.openstack.org.yaml
2026-03-14 09:55:30.458832 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/novaschedulers.nova.openstack.org.yaml
2026-03-14 09:55:30.458848 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/octaviaamphoracontrollers.octavia.openstack.org.yaml
2026-03-14 09:55:30.458864 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/octaviaapis.octavia.openstack.org.yaml
2026-03-14 09:55:30.458880 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/octaviarsyslogs.octavia.openstack.org.yaml
2026-03-14 09:55:30.458896 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/octavias.octavia.openstack.org.yaml
2026-03-14 09:55:30.458911 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackbaremetalsets.baremetal.openstack.org.yaml
2026-03-14 09:55:30.458927 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackclients.client.openstack.org.yaml
2026-03-14 09:55:30.459064 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackcontrolplanes.core.openstack.org.yaml
2026-03-14 09:55:30.459108 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackdataplanedeployments.dataplane.openstack.org.yaml
2026-03-14 09:55:30.459131 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackdataplanenodesets.dataplane.openstack.org.yaml
2026-03-14 09:55:30.459149 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackdataplaneservices.dataplane.openstack.org.yaml
2026-03-14 09:55:30.459166 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackprovisionservers.baremetal.openstack.org.yaml
2026-03-14 09:55:30.459182 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstacks.operator.openstack.org.yaml
2026-03-14 09:55:30.459198 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/openstackversions.core.openstack.org.yaml
2026-03-14 09:55:30.459214 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ovncontrollers.ovn.openstack.org.yaml
2026-03-14 09:55:30.459230 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ovndbclusters.ovn.openstack.org.yaml
2026-03-14 09:55:30.459245 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/ovnnorthds.ovn.openstack.org.yaml
2026-03-14 09:55:30.459381 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/placementapis.placement.openstack.org.yaml
2026-03-14 09:55:30.459402 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/rabbitmqclusters.rabbitmq.com.yaml
2026-03-14 09:55:30.459419 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/rabbitmqpolicies.rabbitmq.openstack.org.yaml
2026-03-14 09:55:30.459435 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/rabbitmqs.rabbitmq.openstack.org.yaml
2026-03-14 09:55:30.459450 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/rabbitmqusers.rabbitmq.openstack.org.yaml
2026-03-14 09:55:30.459466 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/rabbitmqvhosts.rabbitmq.openstack.org.yaml
2026-03-14 09:55:30.459481 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/redises.redis.openstack.org.yaml
2026-03-14 09:55:30.459505 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/reservations.network.openstack.org.yaml
2026-03-14 09:55:30.459522 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/swiftproxies.swift.openstack.org.yaml
2026-03-14 09:55:30.459538 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/swiftrings.swift.openstack.org.yaml
2026-03-14 09:55:30.459555 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/swifts.swift.openstack.org.yaml
2026-03-14 09:55:30.459571 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/swiftstorages.swift.openstack.org.yaml
2026-03-14 09:55:30.459588 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/telemetries.telemetry.openstack.org.yaml
2026-03-14 09:55:30.459821 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/tempests.test.openstack.org.yaml
2026-03-14 09:55:30.459846 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/tobikoes.test.openstack.org.yaml
2026-03-14 09:55:30.459864 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/topologies.topology.openstack.org.yaml
2026-03-14 09:55:30.459881 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/transporturls.rabbitmq.openstack.org.yaml
2026-03-14 09:55:30.459897 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/watcherapis.watcher.openstack.org.yaml
2026-03-14 09:55:30.459913 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/watcherappliers.watcher.openstack.org.yaml
2026-03-14 09:55:30.459929 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/watcherdecisionengines.watcher.openstack.org.yaml
2026-03-14 09:55:30.460217 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/crd/watchers.watcher.openstack.org.yaml
2026-03-14 09:55:30.460248 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/csv/
2026-03-14 09:55:30.460270 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/csv/cluster-observability-operator.v1.3.1
2026-03-14 09:55:30.460868 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/csv/kubernetes-nmstate-operator.4.18.0-202603041813
2026-03-14 09:55:30.460901 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/csv/metallb-operator.v4.18.0-202603040208
2026-03-14 09:55:30.460920 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/csv/openstack-operator.v0.5.0
2026-03-14 09:55:30.460937 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/csv/watcher-operator.v0.0.1
2026-03-14 09:55:30.460954 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/ctlplane/
2026-03-14 09:55:30.460970 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/ctlplane/endpoints
2026-03-14 09:55:30.460986 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/ctlplane/services
2026-03-14 09:55:30.461002 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/ctlplane/ovn/
2026-03-14 09:55:30.461038 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/ctlplane/rabbitmq/
2026-03-14 09:55:30.461770 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/dbs/
2026-03-14 09:55:30.461798 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/dbs/watcher-kuttl-default/
2026-03-14 09:55:30.461816 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/dbs/watcher-kuttl-default/openstack-galera-0-openstack_databases.sql.gz
2026-03-14 09:55:30.461833 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/
2026-03-14 09:55:30.461850 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/
2026-03-14 09:55:30.461866 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/all_resources.log
2026-03-14 09:55:30.461883 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/events.log
2026-03-14 09:55:30.461906 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/nad.log
2026-03-14 09:55:30.461923 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pvc.log
2026-03-14 09:55:30.461938 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/buildconfig/
2026-03-14 09:55:30.461954 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/cert/
2026-03-14 09:55:30.461969 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/certificaterequests/
2026-03-14 09:55:30.461985 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/configmaps/
2026-03-14 09:55:30.462001 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/cronjobs/
2026-03-14 09:55:30.462193 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/daemonset/
2026-03-14 09:55:30.462222 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/deployments/
2026-03-14 09:55:30.462240 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/deployments/cert-manager-cainjector.yaml
2026-03-14 09:55:30.462257 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/deployments/cert-manager-webhook.yaml
2026-03-14 09:55:30.462273 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/deployments/cert-manager.yaml
2026-03-14 09:55:30.462289 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/imagestream/
2026-03-14 09:55:30.462305 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/installplans/
2026-03-14 09:55:30.462321 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/issuer/
2026-03-14 09:55:30.462341 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/jobs/
2026-03-14 09:55:30.462356 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/poddisruptionbudgets/
2026-03-14 09:55:30.462372 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/
2026-03-14 09:55:30.462388 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-545d4d4674-jmg2k/
2026-03-14 09:55:30.462404 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-545d4d4674-jmg2k/cert-manager-545d4d4674-jmg2k-describe
2026-03-14 09:55:30.462420 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-545d4d4674-jmg2k/logs/
2026-03-14 09:55:30.463141 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-545d4d4674-jmg2k/logs/cert-manager-controller.log
2026-03-14 09:55:30.463179 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-cainjector-5545bd876-lmlng/
2026-03-14 09:55:30.463200 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-cainjector-5545bd876-lmlng/cert-manager-cainjector-5545bd876-lmlng-describe
2026-03-14 09:55:30.463218 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-cainjector-5545bd876-lmlng/logs/
2026-03-14 09:55:30.463235 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-cainjector-5545bd876-lmlng/logs/cert-manager-cainjector.log
2026-03-14 09:55:30.463252 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-webhook-6888856db4-hfb75/
2026-03-14 09:55:30.463268 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-webhook-6888856db4-hfb75/cert-manager-webhook-6888856db4-hfb75-describe
2026-03-14 09:55:30.463284 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-webhook-6888856db4-hfb75/logs/
2026-03-14 09:55:30.463432 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/pods/cert-manager-webhook-6888856db4-hfb75/logs/cert-manager-webhook.log
2026-03-14 09:55:30.463460 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/replicaset/
2026-03-14 09:55:30.463478 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/replicaset/cert-manager-545d4d4674.yaml
2026-03-14 09:55:30.463494 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/replicaset/cert-manager-cainjector-5545bd876.yaml
2026-03-14 09:55:30.463511 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/replicaset/cert-manager-webhook-6888856db4.yaml
2026-03-14 09:55:30.463527 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/routes/
2026-03-14 09:55:30.463543 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/services/
2026-03-14 09:55:30.463830 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/services/cert-manager-cainjector.yaml
2026-03-14 09:55:30.463855 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/services/cert-manager-webhook.yaml
2026-03-14 09:55:30.463873 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/services/cert-manager.yaml
2026-03-14 09:55:30.463889 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/statefulset/
2026-03-14 09:55:30.463906 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/cert-manager/subscriptions/
2026-03-14 09:55:30.463922 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/
2026-03-14 09:55:30.463938 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/all_resources.log
2026-03-14 09:55:30.463955 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/events.log
2026-03-14 09:55:30.463976 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/nad.log
2026-03-14 09:55:30.463993 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pvc.log
2026-03-14 09:55:30.464009 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/buildconfig/
2026-03-14 09:55:30.464141 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/cert/
2026-03-14 09:55:30.464163 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/certificaterequests/
2026-03-14 09:55:30.464180 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/configmaps/
2026-03-14 09:55:30.464196 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/cronjobs/
2026-03-14 09:55:30.464212 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/
2026-03-14 09:55:30.464228 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/ipaddresspools.metallb.io/
2026-03-14 09:55:30.464244 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/ipaddresspools.metallb.io/ctlplane.yaml
2026-03-14 09:55:30.464260 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/ipaddresspools.metallb.io/designateext.yaml
2026-03-14 09:55:30.464277 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/ipaddresspools.metallb.io/internalapi.yaml
2026-03-14 09:55:30.464293 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/ipaddresspools.metallb.io/storage.yaml
2026-03-14 09:55:30.464309 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/ipaddresspools.metallb.io/tenant.yaml
2026-03-14 09:55:30.464325 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/l2advertisements.metallb.io/
2026-03-14 09:55:30.464431 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/l2advertisements.metallb.io/ctlplane.yaml
2026-03-14 09:55:30.464458 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/l2advertisements.metallb.io/designateext.yaml
2026-03-14 09:55:30.464476 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/l2advertisements.metallb.io/internalapi.yaml
2026-03-14 09:55:30.464493 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/l2advertisements.metallb.io/storage.yaml
2026-03-14 09:55:30.464509 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/l2advertisements.metallb.io/tenant.yaml
2026-03-14 09:55:30.464525 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/metallbs.metallb.io/
2026-03-14 09:55:30.464540 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/crs/metallbs.metallb.io/metallb.yaml
2026-03-14 09:55:30.464556 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/daemonset/
2026-03-14 09:55:30.464572 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/daemonset/frr-k8s.yaml
2026-03-14 09:55:30.464587 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/daemonset/speaker.yaml
2026-03-14 09:55:30.464603 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/deployments/
2026-03-14 09:55:30.464618 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/deployments/controller.yaml
2026-03-14 09:55:30.464634 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/deployments/frr-k8s-webhook-server.yaml
2026-03-14 09:55:30.464649 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/deployments/metallb-operator-controller-manager.yaml
2026-03-14 09:55:30.464738 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/deployments/metallb-operator-webhook-server.yaml
2026-03-14 09:55:30.464763 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/imagestream/
2026-03-14 09:55:30.464779 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/installplans/
2026-03-14 09:55:30.464795 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/installplans/install-cnj8d
2026-03-14 09:55:30.464811 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/issuer/
2026-03-14 09:55:30.464826 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/jobs/
2026-03-14 09:55:30.464842 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/poddisruptionbudgets/
2026-03-14 09:55:30.464857 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/
2026-03-14 09:55:30.464873 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/controller-7bb4cc7c98-pmz25/
2026-03-14 09:55:30.464889 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/controller-7bb4cc7c98-pmz25/controller-7bb4cc7c98-pmz25-describe
2026-03-14 09:55:30.464905 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/controller-7bb4cc7c98-pmz25/logs/
2026-03-14 09:55:30.464920 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/controller-7bb4cc7c98-pmz25/logs/controller.log.gz
2026-03-14 09:55:30.464936 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/controller-7bb4cc7c98-pmz25/logs/kube-rbac-proxy.log
2026-03-14 09:55:30.465088 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-s58l8/
2026-03-14 09:55:30.465112 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-s58l8/frr-k8s-s58l8-describe
2026-03-14 09:55:30.465129 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-s58l8/logs/
2026-03-14 09:55:30.465145 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-s58l8/logs/controller.log
2026-03-14 09:55:30.465161 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-s58l8/logs/cp-frr-files.log
2026-03-14 09:55:30.465177 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-s58l8/logs/cp-metrics.log
2026-03-14 09:55:30.465192 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-s58l8/logs/cp-reloader.log
2026-03-14 09:55:30.465208 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-s58l8/logs/frr-metrics.log
2026-03-14 09:55:30.465223 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-s58l8/logs/frr.log.gz
2026-03-14 09:55:30.465321 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-s58l8/logs/kube-rbac-proxy-frr.log
2026-03-14 09:55:30.465350 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-s58l8/logs/kube-rbac-proxy.log
2026-03-14 09:55:30.465369 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-s58l8/logs/reloader.log
2026-03-14 09:55:30.465385 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-webhook-server-bcc4b6f68-nd87q/
2026-03-14 09:55:30.465401 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-webhook-server-bcc4b6f68-nd87q/frr-k8s-webhook-server-bcc4b6f68-nd87q-describe
2026-03-14 09:55:30.465417 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-webhook-server-bcc4b6f68-nd87q/logs/
2026-03-14 09:55:30.465437 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/frr-k8s-webhook-server-bcc4b6f68-nd87q/logs/frr-k8s-webhook-server.log
2026-03-14 09:55:30.465453 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-controller-manager-6ccbfd57fd-wcxtt/
2026-03-14 09:55:30.465469 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-controller-manager-6ccbfd57fd-wcxtt/metallb-operator-controller-manager-6ccbfd57fd-wcxtt-describe
2026-03-14 09:55:30.465485 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-controller-manager-6ccbfd57fd-wcxtt/logs/
2026-03-14 09:55:30.465579 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-controller-manager-6ccbfd57fd-wcxtt/logs/manager.log
2026-03-14 09:55:30.465605 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-webhook-server-886f6977-6gxlb/
2026-03-14 09:55:30.465622 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-webhook-server-886f6977-6gxlb/metallb-operator-webhook-server-886f6977-6gxlb-describe
2026-03-14 09:55:30.465639 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-webhook-server-886f6977-6gxlb/logs/
2026-03-14 09:55:30.465656 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/metallb-operator-webhook-server-886f6977-6gxlb/logs/webhook-server.log
2026-03-14 09:55:30.465672 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/speaker-npbfj/
2026-03-14 09:55:30.465689 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/speaker-npbfj/speaker-npbfj-describe
2026-03-14 09:55:30.465705 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/speaker-npbfj/logs/
2026-03-14 09:55:30.465722 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/speaker-npbfj/logs/kube-rbac-proxy.log
2026-03-14 09:55:30.465743 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/pods/speaker-npbfj/logs/speaker.log.gz
2026-03-14 09:55:30.465760 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/replicaset/
2026-03-14 09:55:30.466134 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/replicaset/controller-7bb4cc7c98.yaml
2026-03-14 09:55:30.466173 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/replicaset/frr-k8s-webhook-server-bcc4b6f68.yaml
2026-03-14 09:55:30.466193 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/replicaset/metallb-operator-controller-manager-6ccbfd57fd.yaml
2026-03-14 09:55:30.466211 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/replicaset/metallb-operator-webhook-server-886f6977.yaml
2026-03-14 09:55:30.466228 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/routes/
2026-03-14 09:55:30.466245 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/
2026-03-14 09:55:30.466261 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/controller-monitor-service.yaml
2026-03-14 09:55:30.466278 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/frr-k8s-monitor-service.yaml
2026-03-14 09:55:30.466293 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/frr-k8s-webhook-service.yaml
2026-03-14 09:55:30.466309 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/metallb-operator-controller-manager-service.yaml
2026-03-14 09:55:30.466325 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/metallb-operator-webhook-server-service.yaml
2026-03-14 09:55:30.466439 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/metallb-operator-webhook-service.yaml
2026-03-14 09:55:30.466472 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/speaker-monitor-service.yaml
2026-03-14 09:55:30.466490 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/services/webhook-service.yaml
2026-03-14 09:55:30.466506 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/statefulset/
2026-03-14 09:55:30.466522 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/subscriptions/
2026-03-14 09:55:30.466538 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/metallb-system/subscriptions/metallb-operator-sub
2026-03-14 09:55:30.466553 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/
2026-03-14 09:55:30.466568 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/all_resources.log
2026-03-14 09:55:30.466583 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/events.log
2026-03-14 09:55:30.466598 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/nad.log
2026-03-14 09:55:30.466614 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pvc.log
2026-03-14 09:55:30.466629 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/buildconfig/
2026-03-14 09:55:30.466857 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/cert/
2026-03-14 09:55:30.466887 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/certificaterequests/
2026-03-14 09:55:30.466905 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/configmaps/
2026-03-14 09:55:30.466974 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/cronjobs/
2026-03-14 09:55:30.467005 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/daemonset/
2026-03-14 09:55:30.467046 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/deployments/
2026-03-14 09:55:30.467346 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml
2026-03-14 09:55:30.467368 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/deployments/machine-api-operator.yaml
2026-03-14 09:55:30.467386 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/imagestream/
2026-03-14 09:55:30.467402 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/installplans/
2026-03-14 09:55:30.467418 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/issuer/
2026-03-14 09:55:30.467434 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/jobs/
2026-03-14 09:55:30.467450 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/poddisruptionbudgets/
2026-03-14 09:55:30.467466 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/
2026-03-14 09:55:30.467483 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-78cbb6b69f-d7wqv/
2026-03-14 09:55:30.467499 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-78cbb6b69f-d7wqv/control-plane-machine-set-operator-78cbb6b69f-d7wqv-describe
2026-03-14 09:55:30.467515 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-78cbb6b69f-d7wqv/logs/
2026-03-14 09:55:30.467684 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-78cbb6b69f-d7wqv/logs/control-plane-machine-set-operator.log
2026-03-14 09:55:30.467716 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-8gwm6/
2026-03-14 09:55:30.467734 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-8gwm6/machine-api-operator-5694c8668f-8gwm6-describe
2026-03-14 09:55:30.467751 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-8gwm6/logs/
2026-03-14 09:55:30.467767 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-8gwm6/logs/kube-rbac-proxy.log
2026-03-14 09:55:30.467783 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-8gwm6/logs/machine-api-operator.log
2026-03-14 09:55:30.468108 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/replicaset/
2026-03-14 09:55:30.468129 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-78cbb6b69f.yaml
2026-03-14 09:55:30.468147 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/replicaset/machine-api-operator-5694c8668f.yaml
2026-03-14 09:55:30.468163 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/routes/
2026-03-14 09:55:30.468180 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/
2026-03-14 09:55:30.468195 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml
2026-03-14 09:55:30.468211 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml
2026-03-14 09:55:30.468227 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/machine-api-controllers.yaml
2026-03-14 09:55:30.468243 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml
2026-03-14 09:55:30.469077 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml
2026-03-14 09:55:30.469236 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/services/machine-api-operator.yaml
2026-03-14 09:55:30.469258 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/statefulset/
2026-03-14 09:55:30.469277 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-machine-api/subscriptions/
2026-03-14 09:55:30.469294 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/
2026-03-14 09:55:30.469311 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/all_resources.log
2026-03-14 09:55:30.469327 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/events.log
2026-03-14 09:55:30.469343 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/nad.log
2026-03-14 09:55:30.469359 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pvc.log
2026-03-14 09:55:30.469375 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/buildconfig/
2026-03-14 09:55:30.469390 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/cert/
2026-03-14 09:55:30.469496 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/certificaterequests/
2026-03-14 09:55:30.469524 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/configmaps/
2026-03-14 09:55:30.469542 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/cronjobs/
2026-03-14 09:55:30.469557 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/daemonset/
2026-03-14 09:55:30.469578 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/deployments/
2026-03-14 09:55:30.469595 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/deployments/marketplace-operator.yaml
2026-03-14 09:55:30.469612 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/imagestream/
2026-03-14 09:55:30.469629 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/installplans/
2026-03-14 09:55:30.469645 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/issuer/
2026-03-14 09:55:30.469661 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/jobs/
2026-03-14 09:55:30.469751 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/jobs/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874832f3.yaml
2026-03-14 09:55:30.469778 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/jobs/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c166a6a.yaml
2026-03-14 09:55:30.469796 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/jobs/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e56eb0c.yaml
2026-03-14 09:55:30.469813 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/jobs/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f081954b.yaml
2026-03-14 09:55:30.469829 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/poddisruptionbudgets/
2026-03-14 09:55:30.469845 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/
2026-03-14 09:55:30.469956 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde8744sb7r/
2026-03-14 09:55:30.469983 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde8744sb7r/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde8744sb7r-describe
2026-03-14 09:55:30.470009 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde8744sb7r/logs/
2026-03-14 09:55:30.470164 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde8744sb7r/logs/extract.log
2026-03-14 09:55:30.470187 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde8744sb7r/logs/pull.log
2026-03-14 09:55:30.470204 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde8744sb7r/logs/util.log
2026-03-14 09:55:30.470306 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c1tlsz6/
2026-03-14 09:55:30.470337 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c1tlsz6/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c1tlsz6-describe
2026-03-14 09:55:30.470356 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c1tlsz6/logs/
2026-03-14 09:55:30.470373 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c1tlsz6/logs/extract.log
2026-03-14 09:55:30.470389 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c1tlsz6/logs/pull.log
2026-03-14 09:55:30.470405 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c1tlsz6/logs/util.log
2026-03-14 09:55:30.470427 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e57vqd9/
2026-03-14 09:55:30.470573 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e57vqd9/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e57vqd9-describe
2026-03-14 09:55:30.470605 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e57vqd9/logs/
2026-03-14 09:55:30.470623 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e57vqd9/logs/extract.log
2026-03-14 09:55:30.470642 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e57vqd9/logs/pull.log
2026-03-14 09:55:30.470658 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e57vqd9/logs/util.log
2026-03-14 09:55:30.470674 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f08spsnl/
2026-03-14 09:55:30.470690 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f08spsnl/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f08spsnl-describe
2026-03-14 09:55:30.470706 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f08spsnl/logs/
2026-03-14 09:55:30.470722 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f08spsnl/logs/extract.log
2026-03-14 09:55:30.470738 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f08spsnl/logs/pull.log
2026-03-14 09:55:30.470754 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/98629960b44b381d1a86cff1d1439a8df43509c9ad24579158c59d0f08spsnl/logs/util.log
2026-03-14 09:55:30.471190 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/certified-operators-vzmj4/
2026-03-14 09:55:30.471224 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/certified-operators-vzmj4/certified-operators-vzmj4-describe
2026-03-14 09:55:30.471244 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/certified-operators-vzmj4/logs/
2026-03-14 09:55:30.471260 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/certified-operators-vzmj4/logs/extract-content.log
2026-03-14 09:55:30.471277 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/certified-operators-vzmj4/logs/extract-utilities.log
2026-03-14 09:55:30.471293 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/certified-operators-vzmj4/logs/registry-server.log.gz
2026-03-14 09:55:30.471308 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/community-operators-fnwsk/
2026-03-14 09:55:30.471324 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/community-operators-fnwsk/community-operators-fnwsk-describe
2026-03-14 09:55:30.471341 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/community-operators-fnwsk/logs/
2026-03-14 09:55:30.471359 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/community-operators-fnwsk/logs/extract-content.log
2026-03-14 09:55:30.471460 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/community-operators-fnwsk/logs/extract-utilities.log
2026-03-14 09:55:30.471483 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/community-operators-fnwsk/logs/registry-server.log.gz
2026-03-14 09:55:30.471500 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/marketplace-operator-79b997595-7gnpp/
2026-03-14 09:55:30.471516 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/marketplace-operator-79b997595-7gnpp/marketplace-operator-79b997595-7gnpp-describe
2026-03-14 09:55:30.471537 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/marketplace-operator-79b997595-7gnpp/logs/
2026-03-14 09:55:30.471553 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/marketplace-operator-79b997595-7gnpp/logs/marketplace-operator.log
2026-03-14 09:55:30.471569 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-marketplace-6mtml/
2026-03-14 09:55:30.471585 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-marketplace-6mtml/redhat-marketplace-6mtml-describe
2026-03-14 09:55:30.471601 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-marketplace-6mtml/logs/
2026-03-14 09:55:30.471617 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-marketplace-6mtml/logs/extract-content.log
2026-03-14 09:55:30.471760 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-marketplace-6mtml/logs/extract-utilities.log
2026-03-14 09:55:30.471788 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-marketplace-6mtml/logs/registry-server.log
2026-03-14 09:55:30.471807 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-operators-5d92s/
2026-03-14 09:55:30.471824 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-operators-5d92s/redhat-operators-5d92s-describe
2026-03-14 09:55:30.471840 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-operators-5d92s/logs/
2026-03-14 09:55:30.471857 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-operators-5d92s/logs/extract-content.log
2026-03-14 09:55:30.471873 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-operators-5d92s/logs/extract-utilities.log
2026-03-14 09:55:30.471889 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/pods/redhat-operators-5d92s/logs/registry-server.log.gz
2026-03-14 09:55:30.471909 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/replicaset/
2026-03-14 09:55:30.471926 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/replicaset/marketplace-operator-79b997595.yaml
2026-03-14 09:55:30.472225 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/routes/
2026-03-14 09:55:30.472260 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/services/
2026-03-14 09:55:30.472280 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/services/certified-operators.yaml
2026-03-14 09:55:30.472297 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/services/community-operators.yaml
2026-03-14 09:55:30.472317 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/services/marketplace-operator-metrics.yaml
2026-03-14 09:55:30.472334 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/services/redhat-marketplace.yaml
2026-03-14 09:55:30.472350 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/services/redhat-operators.yaml
2026-03-14 09:55:30.472367 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/statefulset/
2026-03-14 09:55:30.472383 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-marketplace/subscriptions/
2026-03-14 09:55:30.472399 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/
2026-03-14 09:55:30.472415 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/all_resources.log
2026-03-14 09:55:30.472604 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/events.log
2026-03-14 09:55:30.472630 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/nad.log
2026-03-14 09:55:30.472647 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pvc.log
2026-03-14 09:55:30.472664 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/buildconfig/
2026-03-14 09:55:30.472680 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/cert/
2026-03-14 09:55:30.472696 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/certificaterequests/
2026-03-14 09:55:30.472712 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/configmaps/
2026-03-14 09:55:30.472728 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/cronjobs/
2026-03-14 09:55:30.472744 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/daemonset/
2026-03-14 09:55:30.472759 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/daemonset/nmstate-handler.yaml
2026-03-14 09:55:30.472904 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/deployments/
2026-03-14 09:55:30.472927 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/deployments/nmstate-console-plugin.yaml
2026-03-14 09:55:30.472944 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/deployments/nmstate-metrics.yaml
2026-03-14 09:55:30.472961 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/deployments/nmstate-operator.yaml
2026-03-14 09:55:30.472977 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/deployments/nmstate-webhook.yaml
2026-03-14 09:55:30.472993 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/imagestream/
2026-03-14 09:55:30.473027 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/installplans/
2026-03-14 09:55:30.473063 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/installplans/install-dbl44
2026-03-14 09:55:30.473080 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/issuer/
2026-03-14 09:55:30.473097 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/jobs/
2026-03-14 09:55:30.473419 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/poddisruptionbudgets/
2026-03-14 09:55:30.473449 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/poddisruptionbudgets/nmstate-webhook.yaml
2026-03-14 09:55:30.473468 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/
2026-03-14 09:55:30.473485 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-console-plugin-86f58fcf4-6w69h/
2026-03-14 09:55:30.473502 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-console-plugin-86f58fcf4-6w69h/nmstate-console-plugin-86f58fcf4-6w69h-describe
2026-03-14 09:55:30.473519 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-console-plugin-86f58fcf4-6w69h/logs/
2026-03-14 09:55:30.473535 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-console-plugin-86f58fcf4-6w69h/logs/nmstate-console-plugin.log
2026-03-14 09:55:30.473551 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-handler-jz8xl/
2026-03-14 09:55:30.473753 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-handler-jz8xl/nmstate-handler-jz8xl-describe
2026-03-14 09:55:30.473772 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-handler-jz8xl/logs/
2026-03-14 09:55:30.473793 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-handler-jz8xl/logs/nmstate-handler.log
2026-03-14 09:55:30.473943 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-metrics-9b8c8685d-wggcs/
2026-03-14 09:55:30.473971 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-metrics-9b8c8685d-wggcs/nmstate-metrics-9b8c8685d-wggcs-describe
2026-03-14 09:55:30.473989 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-metrics-9b8c8685d-wggcs/logs/
2026-03-14 09:55:30.474006 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-metrics-9b8c8685d-wggcs/logs/kube-rbac-proxy.log
2026-03-14 09:55:30.474259 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-metrics-9b8c8685d-wggcs/logs/nmstate-metrics.log
2026-03-14 09:55:30.474287 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-operator-796d4cfff4-scdwq/
2026-03-14 09:55:30.474306 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-operator-796d4cfff4-scdwq/nmstate-operator-796d4cfff4-scdwq-describe
2026-03-14 09:55:30.474323 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-operator-796d4cfff4-scdwq/logs/
2026-03-14 09:55:30.474469 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-operator-796d4cfff4-scdwq/logs/nmstate-operator.log
2026-03-14 09:55:30.474496 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-webhook-5f558f5558-wdzfb/
2026-03-14 09:55:30.474514 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-webhook-5f558f5558-wdzfb/nmstate-webhook-5f558f5558-wdzfb-describe
2026-03-14 09:55:30.474531 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-webhook-5f558f5558-wdzfb/logs/
2026-03-14 09:55:30.474548 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/pods/nmstate-webhook-5f558f5558-wdzfb/logs/nmstate-webhook.log
2026-03-14 09:55:30.474568 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/replicaset/
2026-03-14 09:55:30.474668 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/replicaset/nmstate-console-plugin-86f58fcf4.yaml
2026-03-14 09:55:30.474695 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/replicaset/nmstate-metrics-9b8c8685d.yaml
2026-03-14 09:55:30.474713 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/replicaset/nmstate-operator-796d4cfff4.yaml
2026-03-14 09:55:30.474730 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/replicaset/nmstate-webhook-5f558f5558.yaml
2026-03-14 09:55:30.474814 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/routes/
2026-03-14 09:55:30.474839 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/services/
2026-03-14 09:55:30.474856 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/services/nmstate-console-plugin.yaml
2026-03-14 09:55:30.474873 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/services/nmstate-monitor.yaml
2026-03-14 09:55:30.475025 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/services/nmstate-webhook.yaml
2026-03-14 09:55:30.475080 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/statefulset/
2026-03-14 09:55:30.475188 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/subscriptions/
2026-03-14 09:55:30.475213 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-nmstate/subscriptions/kubernetes-nmstate-operator
2026-03-14 09:55:30.475231 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/
2026-03-14 09:55:30.475252 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/all_resources.log
2026-03-14 09:55:30.475269 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/events.log
2026-03-14 09:55:30.475285 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/nad.log
2026-03-14 09:55:30.475406 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pvc.log
2026-03-14 09:55:30.475425 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/buildconfig/
2026-03-14 09:55:30.475442 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/cert/
2026-03-14 09:55:30.475458 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/certificaterequests/
2026-03-14 09:55:30.475475 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/configmaps/
2026-03-14 09:55:30.475491 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/cronjobs/
2026-03-14 09:55:30.475507 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/daemonset/
2026-03-14 09:55:30.475523 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/deployments/
2026-03-14 09:55:30.475539 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/deployments/obo-prometheus-operator-admission-webhook.yaml
2026-03-14 09:55:30.475555 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/deployments/obo-prometheus-operator.yaml
2026-03-14 09:55:30.475572 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/deployments/observability-operator.yaml
2026-03-14 09:55:30.475588 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/deployments/observability-ui-dashboards.yaml
2026-03-14 09:55:30.475747 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/deployments/perses-operator.yaml
2026-03-14 09:55:30.475775 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/imagestream/
2026-03-14 09:55:30.475793 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/installplans/
2026-03-14 09:55:30.475809 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/installplans/install-c2r6x
2026-03-14 09:55:30.475825 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/issuer/
2026-03-14 09:55:30.475840 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/jobs/
2026-03-14 09:55:30.475856 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/poddisruptionbudgets/
2026-03-14 09:55:30.475872 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/poddisruptionbudgets/obo-prometheus-operator-admission-webhook.yaml
2026-03-14 09:55:30.475887 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/
2026-03-14 09:55:30.475903 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-68bc856cb9-259ww/
2026-03-14 09:55:30.475919 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-68bc856cb9-259ww/obo-prometheus-operator-68bc856cb9-259ww-describe
2026-03-14 09:55:30.475935 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-68bc856cb9-259ww/logs/
2026-03-14 09:55:30.475951 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-68bc856cb9-259ww/logs/prometheus-operator.log
2026-03-14 09:55:30.476274 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-6b9c87565f-llzfw/
2026-03-14 09:55:30.476315 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-6b9c87565f-llzfw/obo-prometheus-operator-admission-webhook-6b9c87565f-llzfw-describe
2026-03-14 09:55:30.476336 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-6b9c87565f-llzfw/logs/
2026-03-14 09:55:30.476354 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-6b9c87565f-llzfw/logs/prometheus-operator-admission-webhook.log
2026-03-14 09:55:30.476371 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-6b9c87565f-wt77z/
2026-03-14 09:55:30.476387 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-6b9c87565f-wt77z/obo-prometheus-operator-admission-webhook-6b9c87565f-wt77z-describe
2026-03-14 09:55:30.476543 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-6b9c87565f-wt77z/logs/
2026-03-14 09:55:30.476569 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-6b9c87565f-wt77z/logs/prometheus-operator-admission-webhook.log
2026-03-14 09:55:30.476588 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-operator-59bdc8b94-q6fzl/
2026-03-14 09:55:30.476604 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-operator-59bdc8b94-q6fzl/observability-operator-59bdc8b94-q6fzl-describe
2026-03-14 09:55:30.476621 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-operator-59bdc8b94-q6fzl/logs/
2026-03-14 09:55:30.476637 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-operator-59bdc8b94-q6fzl/logs/operator.log
2026-03-14 09:55:30.476652 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-ui-dashboards-66cbf594b5-hkwjk/
2026-03-14 09:55:30.476677 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-ui-dashboards-66cbf594b5-hkwjk/observability-ui-dashboards-66cbf594b5-hkwjk-describe
2026-03-14 09:55:30.476694 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-ui-dashboards-66cbf594b5-hkwjk/logs/
2026-03-14 09:55:30.476710 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/observability-ui-dashboards-66cbf594b5-hkwjk/logs/observability-ui-dashboards.log
2026-03-14 09:55:30.476795 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/perses-operator-5bf474d74f-2vqmj/
2026-03-14 09:55:30.476823 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/perses-operator-5bf474d74f-2vqmj/perses-operator-5bf474d74f-2vqmj-describe
2026-03-14 09:55:30.476841 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/perses-operator-5bf474d74f-2vqmj/logs/
2026-03-14 09:55:30.476858 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/pods/perses-operator-5bf474d74f-2vqmj/logs/perses-operator.log
2026-03-14 09:55:30.476874 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/replicaset/
2026-03-14 09:55:30.476890 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/replicaset/obo-prometheus-operator-68bc856cb9.yaml
2026-03-14 09:55:30.476906 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/replicaset/obo-prometheus-operator-admission-webhook-6b9c87565f.yaml
2026-03-14 09:55:30.476922 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/replicaset/observability-operator-59bdc8b94.yaml
2026-03-14 09:55:30.476939 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/replicaset/observability-ui-dashboards-66cbf594b5.yaml
2026-03-14 09:55:30.477187 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/replicaset/perses-operator-5bf474d74f.yaml
2026-03-14 09:55:30.477214 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/routes/
2026-03-14 09:55:30.477233 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/services/
2026-03-14 09:55:30.477250 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/services/obo-prometheus-operator-admission-webhook-service.yaml
2026-03-14 09:55:30.477266 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/services/obo-prometheus-operator-admission-webhook.yaml
2026-03-14 09:55:30.477283 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/services/obo-prometheus-operator.yaml
2026-03-14 09:55:30.477300 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/services/observability-operator.yaml
2026-03-14 09:55:30.477316 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/services/observability-ui-dashboards.yaml
2026-03-14 09:55:30.477334 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/statefulset/
2026-03-14 09:55:30.477353 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/subscriptions/
2026-03-14 09:55:30.477370 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/subscriptions/cluster-observability-operator
2026-03-14 09:55:30.477387 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openshift-operators/subscriptions/observability-operator
2026-03-14 09:55:30.477403 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/
2026-03-14 09:55:30.477419 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/all_resources.log
2026-03-14 09:55:30.477435 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/events.log
2026-03-14 09:55:30.477455 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/nad.log
2026-03-14 09:55:30.477471 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pvc.log
2026-03-14 09:55:30.477486 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/buildconfig/
2026-03-14 09:55:30.477502 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/
2026-03-14 09:55:30.477517 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/barbican-operator-metrics-certs.yaml
2026-03-14 09:55:30.477974 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/cinder-operator-metrics-certs.yaml
2026-03-14 09:55:30.477995 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/designate-operator-metrics-certs.yaml
2026-03-14 09:55:30.478024 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/glance-operator-metrics-certs.yaml
2026-03-14 09:55:30.478060 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/heat-operator-metrics-certs.yaml
2026-03-14 09:55:30.478079 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/horizon-operator-metrics-certs.yaml
2026-03-14 09:55:30.478095 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/infra-operator-metrics-certs.yaml
2026-03-14 09:55:30.478110 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/infra-operator-serving-cert.yaml
2026-03-14 09:55:30.478126 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/ironic-operator-metrics-certs.yaml
2026-03-14 09:55:30.478276 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/keystone-operator-metrics-certs.yaml
2026-03-14 09:55:30.478306 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/manila-operator-metrics-certs.yaml
2026-03-14 09:55:30.478329 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/mariadb-operator-metrics-certs.yaml
2026-03-14 09:55:30.478346 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/neutron-operator-metrics-certs.yaml
2026-03-14 09:55:30.478362 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/nova-operator-metrics-certs.yaml
2026-03-14 09:55:30.478378 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/octavia-operator-metrics-certs.yaml
2026-03-14 09:55:30.478394 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/openstack-baremetal-operator-metrics-certs.yaml
2026-03-14 09:55:30.478410 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/openstack-baremetal-operator-serving-cert.yaml
2026-03-14 09:55:30.478426 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/openstack-operator-metrics-certs.yaml
2026-03-14 09:55:30.478442 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/openstack-operator-serving-cert.yaml
2026-03-14 09:55:30.478614 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/ovn-operator-metrics-certs.yaml
2026-03-14 09:55:30.478639 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/placement-operator-metrics-certs.yaml
2026-03-14 09:55:30.478657 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/swift-operator-metrics-certs.yaml
2026-03-14 09:55:30.478673 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/telemetry-operator-metrics-certs.yaml
2026-03-14 09:55:30.478690 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/test-operator-metrics-certs.yaml
2026-03-14 09:55:30.478710 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cert/watcher-operator-metrics-certs.yaml
2026-03-14 09:55:30.478727 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/
2026-03-14 09:55:30.478743 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/barbican-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.478760 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/cinder-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.478775 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/designate-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.478791 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/glance-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.479209 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/heat-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.479243 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/horizon-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.479263 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/infra-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.479280 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/infra-operator-serving-cert-1.yaml
2026-03-14 09:55:30.479296 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/ironic-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.479312 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/keystone-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.479328 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/manila-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.479343 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/mariadb-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.479364 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/neutron-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.479380 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/nova-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.479498 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/octavia-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.479531 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/openstack-baremetal-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.479550 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/openstack-baremetal-operator-serving-cert-1.yaml
2026-03-14 09:55:30.479566 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/openstack-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.479582 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/openstack-operator-serving-cert-1.yaml
2026-03-14 09:55:30.479598 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/ovn-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.479614 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/placement-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.479630 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/swift-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.479645 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/telemetry-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.479661 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/test-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.479677 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/certificaterequests/watcher-operator-metrics-certs-1.yaml
2026-03-14 09:55:30.479698 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/configmaps/
2026-03-14 09:55:30.479714 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/cronjobs/
2026-03-14 09:55:30.479730 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/crs/
2026-03-14 09:55:30.479945 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/crs/openstacks.operator.openstack.org/
2026-03-14 09:55:30.479968 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/crs/openstacks.operator.openstack.org/openstack.yaml
2026-03-14 09:55:30.479985 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/daemonset/
2026-03-14 09:55:30.480003 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/
2026-03-14 09:55:30.480050 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/barbican-operator-controller-manager.yaml
2026-03-14 09:55:30.480071 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/cinder-operator-controller-manager.yaml
2026-03-14 09:55:30.480088 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/designate-operator-controller-manager.yaml
2026-03-14 09:55:30.480105 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/glance-operator-controller-manager.yaml
2026-03-14 09:55:30.480239 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/heat-operator-controller-manager.yaml
2026-03-14 09:55:30.480267 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/horizon-operator-controller-manager.yaml
2026-03-14 09:55:30.480285 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/infra-operator-controller-manager.yaml
2026-03-14 09:55:30.480305 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/ironic-operator-controller-manager.yaml
2026-03-14 09:55:30.480322 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/keystone-operator-controller-manager.yaml
2026-03-14 09:55:30.480338 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/manila-operator-controller-manager.yaml
2026-03-14 09:55:30.480354 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/mariadb-operator-controller-manager.yaml
2026-03-14 09:55:30.480369 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/neutron-operator-controller-manager.yaml
2026-03-14 09:55:30.480388 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/nova-operator-controller-manager.yaml
2026-03-14 09:55:30.480406 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/octavia-operator-controller-manager.yaml
2026-03-14 09:55:30.480423 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/openstack-baremetal-operator-controller-manager.yaml
2026-03-14 09:55:30.480439 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/openstack-operator-controller-init.yaml
2026-03-14 09:55:30.480456 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/openstack-operator-controller-manager.yaml
2026-03-14 09:55:30.480472 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/ovn-operator-controller-manager.yaml
2026-03-14 09:55:30.480487 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/placement-operator-controller-manager.yaml
2026-03-14 09:55:30.480503 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/rabbitmq-cluster-operator-manager.yaml
2026-03-14 09:55:30.480523 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/swift-operator-controller-manager.yaml
2026-03-14 09:55:30.480538 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/telemetry-operator-controller-manager.yaml
2026-03-14 09:55:30.481039 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/test-operator-controller-manager.yaml
2026-03-14 09:55:30.481078 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/deployments/watcher-operator-controller-manager.yaml
2026-03-14 09:55:30.481101 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/imagestream/
2026-03-14 09:55:30.481123 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/installplans/
2026-03-14 09:55:30.481144 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/installplans/install-bdgww
2026-03-14 09:55:30.481164 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/installplans/install-ww4hp
2026-03-14 09:55:30.481272 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/
2026-03-14 09:55:30.481303 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/barbican-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481326 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/cinder-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481347 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/designate-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481368 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/glance-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481388 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/heat-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481532 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/horizon-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481562 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/infra-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481581 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/ironic-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481597 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/keystone-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481613 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/manila-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481630 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/mariadb-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481646 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/neutron-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481661 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/nova-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481676 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/octavia-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481692 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/openstack-baremetal-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481708 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/openstack-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481724 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/ovn-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481892 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/placement-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481925 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/rabbitmq-cluster-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481944 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/swift-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481960 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/telemetry-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481976 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/test-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.481992 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/issuer/watcher-operator-selfsigned-issuer.yaml
2026-03-14 09:55:30.482008 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/jobs/
2026-03-14 09:55:30.482058 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/jobs/62d0bed58019ccc1d3626e895c0dbede7faaf3676e025b9f97c0e0a616b757f.yaml
2026-03-14 09:55:30.482076 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/jobs/632ec92be58b5de1327ed8930598ec28f26f424a3aad5e3b3c0e97c82c3006d.yaml
2026-03-14 09:55:30.482093 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/poddisruptionbudgets/
2026-03-14 09:55:30.482109 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/
2026-03-14 09:55:30.482124 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/62d0bed58019ccc1d3626e895c0dbede7faaf3676e025b9f97c0e0a616vmnnb/
2026-03-14 09:55:30.482141 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/62d0bed58019ccc1d3626e895c0dbede7faaf3676e025b9f97c0e0a616vmnnb/62d0bed58019ccc1d3626e895c0dbede7faaf3676e025b9f97c0e0a616vmnnb-describe
2026-03-14 09:55:30.482157 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/62d0bed58019ccc1d3626e895c0dbede7faaf3676e025b9f97c0e0a616vmnnb/logs/
2026-03-14 09:55:30.482177 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/62d0bed58019ccc1d3626e895c0dbede7faaf3676e025b9f97c0e0a616vmnnb/logs/extract.log
2026-03-14 09:55:30.482194 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/62d0bed58019ccc1d3626e895c0dbede7faaf3676e025b9f97c0e0a616vmnnb/logs/pull.log
2026-03-14 09:55:30.482209 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/62d0bed58019ccc1d3626e895c0dbede7faaf3676e025b9f97c0e0a616vmnnb/logs/util.log
2026-03-14 09:55:30.482225 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/632ec92be58b5de1327ed8930598ec28f26f424a3aad5e3b3c0e97c82cpbhtr/
2026-03-14 09:55:30.482241 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/632ec92be58b5de1327ed8930598ec28f26f424a3aad5e3b3c0e97c82cpbhtr/632ec92be58b5de1327ed8930598ec28f26f424a3aad5e3b3c0e97c82cpbhtr-describe
2026-03-14 09:55:30.482257 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/632ec92be58b5de1327ed8930598ec28f26f424a3aad5e3b3c0e97c82cpbhtr/logs/
2026-03-14 09:55:30.482274 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/632ec92be58b5de1327ed8930598ec28f26f424a3aad5e3b3c0e97c82cpbhtr/logs/extract.log
2026-03-14 09:55:30.482290 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/632ec92be58b5de1327ed8930598ec28f26f424a3aad5e3b3c0e97c82cpbhtr/logs/pull.log
2026-03-14 09:55:30.482306 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/632ec92be58b5de1327ed8930598ec28f26f424a3aad5e3b3c0e97c82cpbhtr/logs/util.log
2026-03-14 09:55:30.483073 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/barbican-operator-controller-manager-64768694d-m9dwm/
2026-03-14 09:55:30.483109 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/barbican-operator-controller-manager-64768694d-m9dwm/barbican-operator-controller-manager-64768694d-m9dwm-describe
2026-03-14 09:55:30.483129 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/barbican-operator-controller-manager-64768694d-m9dwm/logs/
2026-03-14 09:55:30.483150 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/barbican-operator-controller-manager-64768694d-m9dwm/logs/manager.log.gz
2026-03-14 09:55:30.483168 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/cinder-operator-controller-manager-cb6d66846-b5jzw/
2026-03-14 09:55:30.483184 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/cinder-operator-controller-manager-cb6d66846-b5jzw/cinder-operator-controller-manager-cb6d66846-b5jzw-describe
2026-03-14 09:55:30.483201 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/cinder-operator-controller-manager-cb6d66846-b5jzw/logs/
2026-03-14 09:55:30.483217 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/cinder-operator-controller-manager-cb6d66846-b5jzw/logs/manager.log.gz
2026-03-14 09:55:30.484782 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/designate-operator-controller-manager-9c8c85cd7-8xwbp/
2026-03-14 09:55:30.488408 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/designate-operator-controller-manager-9c8c85cd7-8xwbp/designate-operator-controller-manager-9c8c85cd7-8xwbp-describe
2026-03-14 09:55:30.488441 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/designate-operator-controller-manager-9c8c85cd7-8xwbp/logs/
2026-03-14 09:55:30.488461 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/designate-operator-controller-manager-9c8c85cd7-8xwbp/logs/manager.log
2026-03-14 09:55:30.488480 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/glance-operator-controller-manager-74d565fbd5-c924b/
2026-03-14 09:55:30.488497 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/glance-operator-controller-manager-74d565fbd5-c924b/glance-operator-controller-manager-74d565fbd5-c924b-describe
2026-03-14 09:55:30.488666 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/glance-operator-controller-manager-74d565fbd5-c924b/logs/
2026-03-14 09:55:30.488693 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/glance-operator-controller-manager-74d565fbd5-c924b/logs/manager.log
2026-03-14 09:55:30.488727 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/heat-operator-controller-manager-6d6bd468b-db2v8/
2026-03-14 09:55:30.488746 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/heat-operator-controller-manager-6d6bd468b-db2v8/heat-operator-controller-manager-6d6bd468b-db2v8-describe
2026-03-14 09:55:30.488763 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/heat-operator-controller-manager-6d6bd468b-db2v8/logs/
2026-03-14 09:55:30.488780 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/heat-operator-controller-manager-6d6bd468b-db2v8/logs/manager.log
2026-03-14 09:55:30.488796 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/horizon-operator-controller-manager-5b9475cdd7-586s5/
2026-03-14 09:55:30.488814 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/horizon-operator-controller-manager-5b9475cdd7-586s5/horizon-operator-controller-manager-5b9475cdd7-586s5-describe
2026-03-14 09:55:30.488830 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/horizon-operator-controller-manager-5b9475cdd7-586s5/logs/
2026-03-14 09:55:30.488847 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/horizon-operator-controller-manager-5b9475cdd7-586s5/logs/manager.log
2026-03-14 09:55:30.488863 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/infra-operator-controller-manager-fbfb5bd65-v7cqm/
2026-03-14 09:55:30.489368 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/infra-operator-controller-manager-fbfb5bd65-v7cqm/infra-operator-controller-manager-fbfb5bd65-v7cqm-describe
2026-03-14 09:55:30.489397 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/infra-operator-controller-manager-fbfb5bd65-v7cqm/logs/
2026-03-14 09:55:30.489415 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/infra-operator-controller-manager-fbfb5bd65-v7cqm/logs/manager.log
2026-03-14 09:55:30.489439 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ironic-operator-controller-manager-bf6b7fd8c-56k2s/
2026-03-14 09:55:30.489456 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ironic-operator-controller-manager-bf6b7fd8c-56k2s/ironic-operator-controller-manager-bf6b7fd8c-56k2s-describe
2026-03-14 09:55:30.489473 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ironic-operator-controller-manager-bf6b7fd8c-56k2s/logs/
2026-03-14 09:55:30.489636 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ironic-operator-controller-manager-bf6b7fd8c-56k2s/logs/manager.log
2026-03-14 09:55:30.489660 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/keystone-operator-controller-manager-68f8d496f8-4knlv/
2026-03-14 09:55:30.489678 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/keystone-operator-controller-manager-68f8d496f8-4knlv/keystone-operator-controller-manager-68f8d496f8-4knlv-describe
2026-03-14 09:55:30.489695 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/keystone-operator-controller-manager-68f8d496f8-4knlv/logs/
2026-03-14 09:55:30.489712 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/keystone-operator-controller-manager-68f8d496f8-4knlv/logs/manager.log
2026-03-14 09:55:30.489728 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/manila-operator-controller-manager-6f6f57b9b6-9c7lm/
2026-03-14 09:55:30.489744 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/manila-operator-controller-manager-6f6f57b9b6-9c7lm/manila-operator-controller-manager-6f6f57b9b6-9c7lm-describe
2026-03-14 09:55:30.489759 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/manila-operator-controller-manager-6f6f57b9b6-9c7lm/logs/
2026-03-14 09:55:30.489775 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/manila-operator-controller-manager-6f6f57b9b6-9c7lm/logs/manager.log
2026-03-14 09:55:30.489795 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/mariadb-operator-controller-manager-744456f686-nnfnk/
2026-03-14 09:55:30.489811 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/mariadb-operator-controller-manager-744456f686-nnfnk/mariadb-operator-controller-manager-744456f686-nnfnk-describe
2026-03-14 09:55:30.489827 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/mariadb-operator-controller-manager-744456f686-nnfnk/logs/
2026-03-14 09:55:30.489842 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/mariadb-operator-controller-manager-744456f686-nnfnk/logs/manager.log
2026-03-14 09:55:30.490046 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/neutron-operator-controller-manager-645c9f6488-9bn7h/
2026-03-14 09:55:30.490069 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/neutron-operator-controller-manager-645c9f6488-9bn7h/neutron-operator-controller-manager-645c9f6488-9bn7h-describe
2026-03-14 09:55:30.490087 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/neutron-operator-controller-manager-645c9f6488-9bn7h/logs/
2026-03-14 09:55:30.490104 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/neutron-operator-controller-manager-645c9f6488-9bn7h/logs/manager.log
2026-03-14 09:55:30.490120 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/nova-operator-controller-manager-58ff56fcc7-9kjv4/
2026-03-14 09:55:30.490135 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/nova-operator-controller-manager-58ff56fcc7-9kjv4/nova-operator-controller-manager-58ff56fcc7-9kjv4-describe
2026-03-14 09:55:30.490151 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/nova-operator-controller-manager-58ff56fcc7-9kjv4/logs/
2026-03-14 09:55:30.490167 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/nova-operator-controller-manager-58ff56fcc7-9kjv4/logs/manager.log
2026-03-14 09:55:30.490316 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/octavia-operator-controller-manager-7cf9f49d6-qwdtd/
2026-03-14 09:55:30.490335 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/octavia-operator-controller-manager-7cf9f49d6-qwdtd/octavia-operator-controller-manager-7cf9f49d6-qwdtd-describe
2026-03-14 09:55:30.490351 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/octavia-operator-controller-manager-7cf9f49d6-qwdtd/logs/
2026-03-14 09:55:30.490368 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/octavia-operator-controller-manager-7cf9f49d6-qwdtd/logs/manager.log
2026-03-14 09:55:30.490384 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-baremetal-operator-controller-manager-c5677dc5d-xbdph/
2026-03-14 09:55:30.490503 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-baremetal-operator-controller-manager-c5677dc5d-xbdph/openstack-baremetal-operator-controller-manager-c5677dc5d-xbdph-describe
2026-03-14 09:55:30.490525 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-baremetal-operator-controller-manager-c5677dc5d-xbdph/logs/
2026-03-14 09:55:30.490542 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-baremetal-operator-controller-manager-c5677dc5d-xbdph/logs/manager.log
2026-03-14 09:55:30.490558 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-controller-manager-59b5586c67-c9k5q/
2026-03-14 09:55:30.490574 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-controller-manager-59b5586c67-c9k5q/openstack-operator-controller-manager-59b5586c67-c9k5q-describe
2026-03-14 09:55:30.490590 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-controller-manager-59b5586c67-c9k5q/logs/
2026-03-14 09:55:30.490606 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-controller-manager-59b5586c67-c9k5q/logs/manager.log.gz
2026-03-14 09:55:30.490627 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-index-kgjkt/
2026-03-14 09:55:30.490644 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-index-kgjkt/openstack-operator-index-kgjkt-describe
2026-03-14 09:55:30.490660 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-index-kgjkt/logs/
2026-03-14 09:55:30.490677 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/openstack-operator-index-kgjkt/logs/registry-server.log
2026-03-14 09:55:30.490693 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ovn-operator-controller-manager-848d74f969-6n97h/
2026-03-14 09:55:30.490709 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ovn-operator-controller-manager-848d74f969-6n97h/ovn-operator-controller-manager-848d74f969-6n97h-describe
2026-03-14 09:55:30.490726 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ovn-operator-controller-manager-848d74f969-6n97h/logs/
2026-03-14 09:55:30.490742 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/ovn-operator-controller-manager-848d74f969-6n97h/logs/manager.log
2026-03-14 09:55:30.490758 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/placement-operator-controller-manager-b5c469fd-bffxq/
2026-03-14 09:55:30.490775 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/placement-operator-controller-manager-b5c469fd-bffxq/placement-operator-controller-manager-b5c469fd-bffxq-describe
2026-03-14 09:55:30.490791 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/placement-operator-controller-manager-b5c469fd-bffxq/logs/
2026-03-14 09:55:30.491146 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/placement-operator-controller-manager-b5c469fd-bffxq/logs/manager.log
2026-03-14 09:55:30.491182 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/rabbitmq-cluster-operator-manager-668c99d594-vppb7/
2026-03-14 09:55:30.491206 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/rabbitmq-cluster-operator-manager-668c99d594-vppb7/rabbitmq-cluster-operator-manager-668c99d594-vppb7-describe
2026-03-14 09:55:30.491223 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/rabbitmq-cluster-operator-manager-668c99d594-vppb7/logs/
2026-03-14 09:55:30.491240 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/rabbitmq-cluster-operator-manager-668c99d594-vppb7/logs/operator.log
2026-03-14 09:55:30.491257 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/swift-operator-controller-manager-7f7469dbc6-6djhw/
2026-03-14 09:55:30.491399 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/swift-operator-controller-manager-7f7469dbc6-6djhw/swift-operator-controller-manager-7f7469dbc6-6djhw-describe
2026-03-14 09:55:30.491422 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/swift-operator-controller-manager-7f7469dbc6-6djhw/logs/
2026-03-14 09:55:30.491440 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/swift-operator-controller-manager-7f7469dbc6-6djhw/logs/manager.log
2026-03-14 09:55:30.491457 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/telemetry-operator-controller-manager-6646df7cdb-tspwx/
2026-03-14 09:55:30.491472 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/telemetry-operator-controller-manager-6646df7cdb-tspwx/telemetry-operator-controller-manager-6646df7cdb-tspwx-describe
2026-03-14 09:55:30.491488 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/telemetry-operator-controller-manager-6646df7cdb-tspwx/logs/
2026-03-14 09:55:30.491504 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/telemetry-operator-controller-manager-6646df7cdb-tspwx/logs/manager.log.gz
2026-03-14 09:55:30.491519 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/test-operator-controller-manager-8467ccb4c8-r9dtz/
2026-03-14 09:55:30.491653 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/test-operator-controller-manager-8467ccb4c8-r9dtz/test-operator-controller-manager-8467ccb4c8-r9dtz-describe
2026-03-14 09:55:30.491685 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/test-operator-controller-manager-8467ccb4c8-r9dtz/logs/
2026-03-14 09:55:30.491703 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/test-operator-controller-manager-8467ccb4c8-r9dtz/logs/manager.log
2026-03-14 09:55:30.491721 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-controller-manager-788cc4b948-xgdnd/
2026-03-14 09:55:30.491738 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-controller-manager-788cc4b948-xgdnd/watcher-operator-controller-manager-788cc4b948-xgdnd-describe
2026-03-14 09:55:30.491860 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-controller-manager-788cc4b948-xgdnd/logs/
2026-03-14 09:55:30.491885 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-controller-manager-788cc4b948-xgdnd/logs/manager.log.gz
2026-03-14 09:55:30.491903 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-index-jl5p2/
2026-03-14 09:55:30.491920 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-index-jl5p2/watcher-operator-index-jl5p2-describe
2026-03-14 09:55:30.491937 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-index-jl5p2/logs/
2026-03-14 09:55:30.491953 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/pods/watcher-operator-index-jl5p2/logs/registry-server.log
2026-03-14 09:55:30.491970 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/
2026-03-14 09:55:30.491986 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/barbican-operator-controller-manager-64768694d.yaml
2026-03-14 09:55:30.492007 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/cinder-operator-controller-manager-cb6d66846.yaml
2026-03-14 09:55:30.492146 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/designate-operator-controller-manager-9c8c85cd7.yaml
2026-03-14 09:55:30.492169 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/glance-operator-controller-manager-74d565fbd5.yaml
2026-03-14 09:55:30.492187 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/heat-operator-controller-manager-6d6bd468b.yaml
2026-03-14 09:55:30.492204 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/horizon-operator-controller-manager-5b9475cdd7.yaml
2026-03-14 09:55:30.492220 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/infra-operator-controller-manager-fbfb5bd65.yaml
2026-03-14 09:55:30.492236 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/ironic-operator-controller-manager-bf6b7fd8c.yaml
2026-03-14 09:55:30.492253 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/keystone-operator-controller-manager-68f8d496f8.yaml
2026-03-14 09:55:30.492269 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/manila-operator-controller-manager-6f6f57b9b6.yaml
2026-03-14 09:55:30.492285 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/mariadb-operator-controller-manager-744456f686.yaml
2026-03-14 09:55:30.492301 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/neutron-operator-controller-manager-645c9f6488.yaml
2026-03-14 09:55:30.492318 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/nova-operator-controller-manager-58ff56fcc7.yaml
2026-03-14 09:55:30.492450 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/octavia-operator-controller-manager-7cf9f49d6.yaml
2026-03-14 09:55:30.492482 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/openstack-baremetal-operator-controller-manager-c5677dc5d.yaml
2026-03-14 09:55:30.492501 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/openstack-operator-controller-init-6ccbf6d758.yaml
2026-03-14 09:55:30.492518 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/openstack-operator-controller-manager-59b5586c67.yaml
2026-03-14 09:55:30.492535 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/ovn-operator-controller-manager-848d74f969.yaml
2026-03-14 09:55:30.492551 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/placement-operator-controller-manager-b5c469fd.yaml
2026-03-14 09:55:30.492566 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/rabbitmq-cluster-operator-manager-668c99d594.yaml
2026-03-14 09:55:30.492583 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/swift-operator-controller-manager-7f7469dbc6.yaml
2026-03-14 09:55:30.492599 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/telemetry-operator-controller-manager-6646df7cdb.yaml
2026-03-14 09:55:30.492614 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/test-operator-controller-manager-8467ccb4c8.yaml
2026-03-14 09:55:30.492630 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/watcher-operator-controller-manager-788cc4b948.yaml
2026-03-14 09:55:30.492646 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/replicaset/watcher-operator-controller-manager-84884ff888.yaml
2026-03-14 09:55:30.492662 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/routes/
2026-03-14 09:55:30.492678 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/
2026-03-14 09:55:30.492698 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/barbican/
2026-03-14 09:55:30.492838 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/barbican/barbican-operator-metrics-server-cert.yaml
2026-03-14 09:55:30.492859 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/cinder/
2026-03-14 09:55:30.492876 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/cinder/cinder-operator-metrics-server-cert.yaml
2026-03-14 09:55:30.492892 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/designate/
2026-03-14 09:55:30.493057 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/designate/designate-operator-metrics-server-cert.yaml
2026-03-14 09:55:30.493090 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/glance/
2026-03-14 09:55:30.493110 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/glance/glance-operator-metrics-server-cert.yaml
2026-03-14 09:55:30.493127 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/heat/
2026-03-14 09:55:30.493144 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/heat/heat-operator-metrics-server-cert.yaml
2026-03-14 09:55:30.493423 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/horizon/
2026-03-14 09:55:30.493451 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/horizon/horizon-operator-metrics-server-cert.yaml
2026-03-14 09:55:30.493469 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/ironic/
2026-03-14 09:55:30.493486 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/ironic/ironic-operator-metrics-server-cert.yaml
2026-03-14 09:55:30.493585 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/keystone/
2026-03-14 09:55:30.493613 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/keystone/keystone-operator-metrics-server-cert.yaml
2026-03-14 09:55:30.493631 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/manila/
2026-03-14 09:55:30.493649 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/manila/manila-operator-metrics-server-cert.yaml
2026-03-14 09:55:30.493666 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/mariadb/
2026-03-14 09:55:30.493683 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/mariadb/mariadb-operator-metrics-server-cert.yaml
2026-03-14 09:55:30.493699 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/neutron/
2026-03-14 09:55:30.493716 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/neutron/neutron-operator-metrics-server-cert.yaml
2026-03-14 09:55:30.493732 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/nova/
2026-03-14 09:55:30.493749 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/nova/nova-operator-metrics-server-cert.yaml
2026-03-14 09:55:30.493765 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/octavia/
2026-03-14 09:55:30.493781 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/octavia/octavia-operator-metrics-server-cert.yaml
2026-03-14 09:55:30.493797 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/ovn/
2026-03-14 09:55:30.493813 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/ovn/ovn-operator-metrics-server-cert.yaml
2026-03-14 09:55:30.493963 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/placement/
2026-03-14 09:55:30.493992 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/placement/placement-operator-metrics-server-cert.yaml
2026-03-14 09:55:30.494009 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/swift/
2026-03-14 09:55:30.494056 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/swift/swift-operator-metrics-server-cert.yaml
2026-03-14 09:55:30.494075 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/telemetry/
2026-03-14 09:55:30.494092 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/telemetry/telemetry-operator-metrics-server-cert.yaml
2026-03-14 09:55:30.494108 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/watcher/
2026-03-14 09:55:30.494124 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/watcher/watcher-operator-controller-manager-service-cert.yaml
2026-03-14 09:55:30.494140 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/secrets/watcher/watcher-operator-metrics-server-cert.yaml
2026-03-14 09:55:30.494156 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/
2026-03-14 09:55:30.494267 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/barbican-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.494299 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/cinder-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.494317 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/designate-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.494334 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/glance-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.494354 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/heat-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.494370 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/horizon-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.494386 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/infra-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.494402 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/infra-operator-webhook-service.yaml
2026-03-14 09:55:30.494417 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/ironic-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.494433 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/keystone-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.494449 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/manila-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.494465 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/mariadb-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.494681 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/neutron-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.494702 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/nova-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.494720 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/octavia-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.494737 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/openstack-baremetal-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.494753 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/openstack-baremetal-operator-webhook-service.yaml
2026-03-14 09:55:30.494775 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/openstack-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.494792 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/openstack-operator-index.yaml
2026-03-14 09:55:30.494808 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/openstack-operator-webhook-service.yaml
2026-03-14 09:55:30.494823 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/ovn-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.494839 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/placement-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.494854 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/rabbitmq-cluster-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.494986 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/swift-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.495039 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/telemetry-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.495060 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/test-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.495077 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/watcher-operator-controller-manager-metrics-service.yaml
2026-03-14 09:55:30.495094 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/watcher-operator-controller-manager-service.yaml
2026-03-14 09:55:30.495336 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/watcher-operator-index.yaml
2026-03-14 09:55:30.495361 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/services/watcher-operator-webhook-service.yaml
2026-03-14 09:55:30.495379 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/statefulset/
2026-03-14 09:55:30.495396 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/subscriptions/
2026-03-14 09:55:30.495413 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/subscriptions/openstack-operator
2026-03-14 09:55:30.495429 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack-operators/subscriptions/watcher-operator
2026-03-14 09:55:30.495445 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/
2026-03-14 09:55:30.495462 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/all_resources.log
2026-03-14 09:55:30.495482 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/events.log
2026-03-14 09:55:30.495499 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/nad.log
2026-03-14 09:55:30.495525 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/pvc.log
2026-03-14 09:55:30.495674 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/buildconfig/
2026-03-14 09:55:30.495697 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/cert/
2026-03-14 09:55:30.495714 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/certificaterequests/
2026-03-14 09:55:30.495731 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/configmaps/
2026-03-14 09:55:30.495747 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/cronjobs/
2026-03-14 09:55:30.495764 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/
2026-03-14 09:55:30.495784 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/
2026-03-14 09:55:30.495801 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/ctlplane.yaml
2026-03-14 09:55:30.495910 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/datacentre.yaml
2026-03-14 09:55:30.495940 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/designate.yaml
2026-03-14 09:55:30.495959 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/designateext.yaml
2026-03-14 09:55:30.495976 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/internalapi.yaml
2026-03-14 09:55:30.495993 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/octavia.yaml
2026-03-14 09:55:30.496010 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/storage.yaml
2026-03-14 09:55:30.496631 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/storagemgmt.yaml
2026-03-14 09:55:30.496664 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/tenant.yaml
2026-03-14 09:55:30.496684 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/daemonset/
2026-03-14 09:55:30.496701 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/deployments/
2026-03-14 09:55:30.496724 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/imagestream/
2026-03-14 09:55:30.496742 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/installplans/
2026-03-14 09:55:30.496763 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/issuer/
2026-03-14 09:55:30.496780 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/jobs/
2026-03-14 09:55:30.496795 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/poddisruptionbudgets/
2026-03-14 09:55:30.496934 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/pods/
2026-03-14 09:55:30.496957 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/pods/-describe
2026-03-14 09:55:30.496975 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/pods/logs/
2026-03-14 09:55:30.496992 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/replicaset/
2026-03-14 09:55:30.497008 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/routes/
2026-03-14 09:55:30.497045 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/services/
2026-03-14 09:55:30.497063 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/statefulset/
2026-03-14 09:55:30.497079 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/openstack/subscriptions/
2026-03-14 09:55:30.497095 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/
2026-03-14 09:55:30.497111 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/all_resources.log
2026-03-14 09:55:30.497127 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/events.log
2026-03-14 09:55:30.497142 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/nad.log
2026-03-14 09:55:30.497277 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pvc.log
2026-03-14 09:55:30.497387 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/buildconfig/
2026-03-14 09:55:30.497410 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/
2026-03-14 09:55:30.497427 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/ceilometer-internal-svc.yaml
2026-03-14 09:55:30.497444 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/cinder-internal-svc.yaml
2026-03-14 09:55:30.497461 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/galera-openstack-svc.yaml
2026-03-14 09:55:30.497477 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/keystone-internal-svc.yaml
2026-03-14 09:55:30.497639 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/keystone-public-route.yaml
2026-03-14 09:55:30.497658 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/keystone-public-svc.yaml
2026-03-14 09:55:30.497675 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/kube-state-metrics-svc.yaml
2026-03-14 09:55:30.497692 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/memcached-mtls.yaml
2026-03-14 09:55:30.497708 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/memcached-svc.yaml
2026-03-14 09:55:30.497724 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/metric-storage-prometheus-svc.yaml
2026-03-14 09:55:30.497740 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/rabbitmq-notifications-svc.yaml
2026-03-14 09:55:30.497755 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/rabbitmq-svc.yaml
2026-03-14 09:55:30.497775 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/rootca-internal.yaml
2026-03-14 09:55:30.497791 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/rootca-libvirt.yaml
2026-03-14 09:55:30.497806 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/rootca-ovn.yaml
2026-03-14 09:55:30.497822 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cert/rootca-public.yaml
2026-03-14 09:55:30.498028 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/
2026-03-14 09:55:30.498053 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/ceilometer-internal-svc-1.yaml
2026-03-14 09:55:30.498071 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/cinder-internal-svc-1.yaml
2026-03-14 09:55:30.498088 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/galera-openstack-svc-1.yaml
2026-03-14 09:55:30.498254 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/keystone-internal-svc-1.yaml
2026-03-14 09:55:30.498275 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/keystone-public-route-1.yaml
2026-03-14 09:55:30.498292 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/keystone-public-svc-1.yaml
2026-03-14 09:55:30.498309 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/kube-state-metrics-svc-1.yaml
2026-03-14 09:55:30.498326 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/memcached-mtls-1.yaml
2026-03-14 09:55:30.498342 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/memcached-svc-1.yaml
2026-03-14 09:55:30.498478 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/metric-storage-prometheus-svc-1.yaml
2026-03-14 09:55:30.498507 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/rabbitmq-notifications-svc-1.yaml
2026-03-14 09:55:30.498526 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/rabbitmq-svc-1.yaml
2026-03-14 09:55:30.498543 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/rootca-internal-1.yaml
2026-03-14 09:55:30.498561 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/rootca-libvirt-1.yaml
2026-03-14 09:55:30.498577 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/rootca-ovn-1.yaml
2026-03-14 09:55:30.498594 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/certificaterequests/rootca-public-1.yaml
2026-03-14 09:55:30.498610 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/
2026-03-14 09:55:30.498626 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/appcred-watcher-pre.yaml
2026-03-14 09:55:30.498643 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/infra-instanceha-config.yaml
2026-03-14 09:55:30.498659 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/prometheus-metric-storage-rulefiles-0.yaml
2026-03-14 09:55:30.498676 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/rabbitmq-config-data.yaml
2026-03-14 09:55:30.498696 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/rabbitmq-notifications-config-data.yaml
2026-03-14 09:55:30.498717 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/rabbitmq-notifications-plugins-conf.yaml
2026-03-14 09:55:30.499058 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/rabbitmq-notifications-server-conf.yaml
2026-03-14 09:55:30.499084 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/rabbitmq-plugins-conf.yaml
2026-03-14 09:55:30.499102 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/configmaps/rabbitmq-server-conf.yaml
2026-03-14 09:55:30.499120 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cronjobs/
2026-03-14 09:55:30.499136 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/cronjobs/keystone-cron.yaml
2026-03-14 09:55:30.499314 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/
2026-03-14 09:55:30.499334 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/alertmanagers.monitoring.rhobs/
2026-03-14 09:55:30.499351 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/alertmanagers.monitoring.rhobs/metric-storage.yaml
2026-03-14 09:55:30.499367 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/ceilometers.telemetry.openstack.org/
2026-03-14 09:55:30.499383 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/ceilometers.telemetry.openstack.org/ceilometer.yaml
2026-03-14 09:55:30.499400 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/galeras.mariadb.openstack.org/
2026-03-14 09:55:30.499416 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/galeras.mariadb.openstack.org/openstack.yaml
2026-03-14 09:55:30.499600 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/keystoneapis.keystone.openstack.org/
2026-03-14 09:55:30.499626 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/keystoneapis.keystone.openstack.org/keystone.yaml
2026-03-14 09:55:30.499643 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/keystoneservices.keystone.openstack.org/
2026-03-14 09:55:30.499659 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/keystoneservices.keystone.openstack.org/ceilometer.yaml
2026-03-14 09:55:30.499675 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/mariadbaccounts.mariadb.openstack.org/
2026-03-14 09:55:30.499796 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/mariadbaccounts.mariadb.openstack.org/keystone.yaml
2026-03-14 09:55:30.499816 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/mariadbaccounts.mariadb.openstack.org/openstack-mariadb-root.yaml
2026-03-14 09:55:30.499833 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/mariadbdatabases.mariadb.openstack.org/
2026-03-14 09:55:30.499849 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/mariadbdatabases.mariadb.openstack.org/keystone.yaml
2026-03-14 09:55:30.499866 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/memcacheds.memcached.openstack.org/
2026-03-14 09:55:30.499882 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/memcacheds.memcached.openstack.org/memcached.yaml
2026-03-14 09:55:30.499897 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/metricstorages.telemetry.openstack.org/
2026-03-14 09:55:30.499913 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/metricstorages.telemetry.openstack.org/metric-storage.yaml
2026-03-14 09:55:30.499929 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/monitoringstacks.monitoring.rhobs/
2026-03-14 09:55:30.499949 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/monitoringstacks.monitoring.rhobs/metric-storage.yaml
2026-03-14 09:55:30.499966 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/openstackclients.client.openstack.org/
2026-03-14 09:55:30.500152 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/openstackclients.client.openstack.org/openstackclient.yaml
2026-03-14 09:55:30.500174 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/openstackcontrolplanes.core.openstack.org/
2026-03-14 09:55:30.500191 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/openstackcontrolplanes.core.openstack.org/openstack.yaml
2026-03-14 09:55:30.500207 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/openstackversions.core.openstack.org/
2026-03-14 09:55:30.500364 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/openstackversions.core.openstack.org/openstack.yaml
2026-03-14 09:55:30.500392 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/prometheuses.monitoring.rhobs/
2026-03-14 09:55:30.500410 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/prometheuses.monitoring.rhobs/metric-storage.yaml
2026-03-14 09:55:30.500426 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/prometheusrules.monitoring.rhobs/
2026-03-14 09:55:30.500443 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/prometheusrules.monitoring.rhobs/metric-storage.yaml
2026-03-14 09:55:30.500459 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/rabbitmqclusters.rabbitmq.com/
2026-03-14 09:55:30.500475 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/rabbitmqclusters.rabbitmq.com/rabbitmq-notifications.yaml
2026-03-14 09:55:30.500491 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/rabbitmqclusters.rabbitmq.com/rabbitmq.yaml
2026-03-14 09:55:30.500511 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/rabbitmqs.rabbitmq.openstack.org/
2026-03-14 09:55:30.500527 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/rabbitmqs.rabbitmq.openstack.org/rabbitmq-notifications.yaml
2026-03-14 09:55:30.500543 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/rabbitmqs.rabbitmq.openstack.org/rabbitmq.yaml
2026-03-14 09:55:30.500559 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/
2026-03-14 09:55:30.500774 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer-compute-prom-exporter-tls.yaml
2026-03-14 09:55:30.500792 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer-compute-prom-exporter.yaml
2026-03-14 09:55:30.500808 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer-ipmi-prom-exporter-tls.yaml
2026-03-14 09:55:30.500825 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer-ipmi-prom-exporter.yaml
2026-03-14 09:55:30.500841 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer.yaml
2026-03-14 09:55:30.500857 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-kepler.yaml
2026-03-14 09:55:30.500873 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ksm.yaml
2026-03-14 09:55:30.500889 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-node-exporter-tls.yaml
2026-03-14 09:55:30.500905 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-node-exporter.yaml
2026-03-14 09:55:30.501072 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-openstack-network-exporter-tls.yaml
2026-03-14 09:55:30.501100 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-openstack-network-exporter.yaml
2026-03-14 09:55:30.501118 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-podman-exporter-tls.yaml
2026-03-14 09:55:30.501135 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-podman-exporter.yaml
2026-03-14 09:55:30.501152 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-rabbitmq-notifications.yaml
2026-03-14 09:55:30.501168 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-rabbitmq.yaml
2026-03-14 09:55:30.501185 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/telemetries.telemetry.openstack.org/
2026-03-14 09:55:30.501201 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/telemetries.telemetry.openstack.org/telemetry.yaml
2026-03-14 09:55:30.501465 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/transporturls.rabbitmq.openstack.org/
2026-03-14 09:55:30.501492 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/transporturls.rabbitmq.openstack.org/ceilometer-notifications-transport.yaml
2026-03-14 09:55:30.501511 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/crs/transporturls.rabbitmq.openstack.org/keystone-keystone-transport.yaml
2026-03-14 09:55:30.501528 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/daemonset/
2026-03-14 09:55:30.501548 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/deployments/
2026-03-14 09:55:30.501565 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/deployments/keystone.yaml
2026-03-14 09:55:30.501581 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/imagestream/
2026-03-14 09:55:30.501598 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/installplans/
2026-03-14 09:55:30.501614 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/issuer/
2026-03-14 09:55:30.501629 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/issuer/rootca-internal.yaml
2026-03-14 09:55:30.501835 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/issuer/rootca-libvirt.yaml
2026-03-14 09:55:30.501861 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/issuer/rootca-ovn.yaml
2026-03-14 09:55:30.501883 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/issuer/rootca-public.yaml
2026-03-14 09:55:30.501904 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/issuer/selfsigned-issuer.yaml
2026-03-14 09:55:30.501926 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/jobs/
2026-03-14 09:55:30.502234 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/poddisruptionbudgets/
2026-03-14 09:55:30.502264 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/poddisruptionbudgets/metric-storage-alertmanager.yaml
2026-03-14 09:55:30.502287 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/
2026-03-14 09:55:30.502308 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/
2026-03-14 09:55:30.502403 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/alertmanager-metric-storage-0-describe
2026-03-14 09:55:30.502428 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/logs/
2026-03-14 09:55:30.502885 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/logs/alertmanager.log
2026-03-14 09:55:30.502906 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/logs/config-reloader.log
2026-03-14 09:55:30.502923 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/logs/init-config-reloader.log
2026-03-14 09:55:30.502940 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/
2026-03-14 09:55:30.502956 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/ceilometer-0-describe
2026-03-14 09:55:30.502972 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/
2026-03-14 09:55:30.502999 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/ceilometer-central-agent.log
2026-03-14 09:55:30.503187 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/ceilometer-notification-agent.log
2026-03-14 09:55:30.503216 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/proxy-httpd.log
2026-03-14 09:55:30.503235 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/sg-core.log
2026-03-14 09:55:30.503251 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/keystone-75c5988f99-28ht4/
2026-03-14 09:55:30.503268 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/keystone-75c5988f99-28ht4/keystone-75c5988f99-28ht4-describe
2026-03-14 09:55:30.503288 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/keystone-75c5988f99-28ht4/logs/
2026-03-14 09:55:30.503305 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/keystone-75c5988f99-28ht4/logs/keystone-api.log
2026-03-14 09:55:30.503321 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/kube-state-metrics-0/
2026-03-14 09:55:30.503337 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/kube-state-metrics-0/kube-state-metrics-0-describe
2026-03-14 09:55:30.503602 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/kube-state-metrics-0/logs/
2026-03-14 09:55:30.503630 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/kube-state-metrics-0/logs/kube-state-metrics.log
2026-03-14 09:55:30.503648 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/memcached-0/
2026-03-14 09:55:30.503665 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/memcached-0/memcached-0-describe
2026-03-14 09:55:30.503681 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/memcached-0/logs/
2026-03-14 09:55:30.503697 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/memcached-0/logs/memcached.log.gz
2026-03-14 09:55:30.503713 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstack-galera-0/
2026-03-14 09:55:30.503729 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstack-galera-0/openstack-galera-0-describe
2026-03-14 09:55:30.503745 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstack-galera-0/logs/
2026-03-14 09:55:30.503937 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstack-galera-0/logs/galera.log
2026-03-14 09:55:30.503967 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstack-galera-0/logs/mysql-bootstrap.log
2026-03-14 09:55:30.503985 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstackclient/
2026-03-14 09:55:30.504002 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstackclient/openstackclient-describe
2026-03-14 09:55:30.504035 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstackclient/logs/
2026-03-14 09:55:30.504053 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/openstackclient/logs/openstackclient.log
2026-03-14 09:55:30.504068 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/
2026-03-14 09:55:30.504084 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/prometheus-metric-storage-0-describe
2026-03-14 09:55:30.504168 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/
2026-03-14 09:55:30.504200 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/config-reloader.log
2026-03-14 09:55:30.504218 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/init-config-reloader.log
2026-03-14 09:55:30.504235 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/prometheus.log
2026-03-14 09:55:30.504251 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/thanos-sidecar.log
2026-03-14 09:55:30.504267 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/
2026-03-14 09:55:30.504287 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/rabbitmq-notifications-server-0-describe
2026-03-14 09:55:30.504304 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/logs/
2026-03-14 09:55:30.504320 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/logs/rabbitmq.log
2026-03-14 09:55:30.504336 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/logs/setup-container.log
2026-03-14 09:55:30.504353 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/
2026-03-14 09:55:30.504515 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/rabbitmq-server-0-describe
2026-03-14 09:55:30.504543 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/logs/
2026-03-14 09:55:30.504561 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/logs/rabbitmq.log
2026-03-14 09:55:30.504578 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/logs/setup-container.log
2026-03-14 09:55:30.504595 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/replicaset/
2026-03-14 09:55:30.504612 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/replicaset/keystone-6b8d58878f.yaml
2026-03-14 09:55:30.504628 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/replicaset/keystone-75c5988f99.yaml
2026-03-14 09:55:30.504645 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/routes/
2026-03-14 09:55:30.504661 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/routes/keystone-public.yaml
2026-03-14 09:55:30.504894 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/
2026-03-14 09:55:30.504928 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/
2026-03-14 09:55:30.504947 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-compute-config-data.yaml
2026-03-14 09:55:30.504964 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-compute-config-data.yaml-ceilometer.conf
2026-03-14 09:55:30.504981 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-compute-config-data.yaml-custom.conf
2026-03-14 09:55:30.505148 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-compute-scripts.yaml
2026-03-14 09:55:30.505170 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml
2026-03-14 09:55:30.505187 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml-ceilometer.conf
2026-03-14 09:55:30.505204 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml-custom.conf
2026-03-14 09:55:30.505220 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml-httpd.conf
2026-03-14 09:55:30.505236 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml-ssl.conf
2026-03-14 09:55:30.505627 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-ipmi-config-data.yaml
2026-03-14 09:55:30.505659 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-ipmi-config-data.yaml-ceilometer.conf
2026-03-14 09:55:30.505682 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-ipmi-config-data.yaml-custom.conf
2026-03-14 09:55:30.505699 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-ipmi-scripts.yaml
2026-03-14 09:55:30.505716 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-scripts.yaml
2026-03-14 09:55:30.505733 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/cert-ceilometer-internal-svc.yaml
2026-03-14 09:55:30.505749 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ceilometer/rabbitmq-transport-url-ceilometer-notifications-transport.yaml
2026-03-14 09:55:30.505766 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/cinder/
2026-03-14 09:55:30.505782 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/cinder/cert-cinder-internal-svc.yaml
2026-03-14 09:55:30.506092 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/cinder/cert-cinder-public-route.yaml
2026-03-14 09:55:30.506128 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/cinder/cert-cinder-public-svc.yaml
2026-03-14 09:55:30.506148 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/
2026-03-14 09:55:30.506165 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/cert-keystone-internal-svc.yaml
2026-03-14 09:55:30.506181 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/cert-keystone-public-route.yaml
2026-03-14 09:55:30.506198 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/cert-keystone-public-svc.yaml
2026-03-14 09:55:30.506218 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone-config-data.yaml
2026-03-14 09:55:30.506235 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone-config-data.yaml-httpd.conf
2026-03-14 09:55:30.506251 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone-config-data.yaml-keystone.conf
2026-03-14 09:55:30.506453 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone-config-data.yaml-ssl.conf
2026-03-14 09:55:30.506477 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone-db-secret.yaml
2026-03-14 09:55:30.506495 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone-scripts.yaml
2026-03-14 09:55:30.506511 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/keystone.yaml
2026-03-14 09:55:30.506527 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/keystone/rabbitmq-transport-url-keystone-keystone-transport.yaml
2026-03-14 09:55:30.506724 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/mariadb/
2026-03-14 09:55:30.506750 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/mariadb/openstack-mariadb-root-db-secret.yaml
2026-03-14 09:55:30.506768 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/
2026-03-14 09:55:30.506785 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/alertmanager-metric-storage-cluster-tls-config.yaml
2026-03-14 09:55:30.506802 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/alertmanager-metric-storage-generated.yaml
2026-03-14 09:55:30.507039 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/alertmanager-metric-storage-tls-assets-0.yaml
2026-03-14 09:55:30.507075 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/alertmanager-metric-storage-web-config.yaml
2026-03-14 09:55:30.507095 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/cert-metric-storage-prometheus-svc.yaml
2026-03-14 09:55:30.507209 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/metric-storage-prometheus-endpoint.yaml
2026-03-14 09:55:30.507230 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/metric-storage-self-scrape.yaml
2026-03-14 09:55:30.507247 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/prometheus-metric-storage-thanos-prometheus-http-client-file.yaml
2026-03-14 09:55:30.507264 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/prometheus-metric-storage-tls-assets-0.yaml
2026-03-14 09:55:30.507280 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/prometheus-metric-storage-web-config.yaml
2026-03-14 09:55:30.507296 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/metric-storage/prometheus-metric-storage.yaml
2026-03-14 09:55:30.507312 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ovn/
2026-03-14 09:55:30.507328 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/ovn/rootca-ovn.yaml
2026-03-14 09:55:30.507343 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/
2026-03-14 09:55:30.507558 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/cert-rabbitmq-notifications-svc.yaml
2026-03-14 09:55:30.507583 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/cert-rabbitmq-svc.yaml
2026-03-14 09:55:30.507606 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-default-user.yaml
2026-03-14 09:55:30.507622 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-default-user.yaml-default_user.conf
2026-03-14 09:55:30.507639 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-erlang-cookie.yaml
2026-03-14 09:55:30.507655 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-notifications-default-user.yaml
2026-03-14 09:55:30.507671 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-notifications-default-user.yaml-default_user.conf
2026-03-14 09:55:30.507687 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-notifications-erlang-cookie.yaml
2026-03-14 09:55:30.507704 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-transport-url-ceilometer-notifications-transport.yaml
2026-03-14 09:55:30.507720 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-transport-url-keystone-keystone-transport.yaml
2026-03-14 09:55:30.507736 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/watcher/
2026-03-14 09:55:30.507919 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/watcher/cert-watcher-internal-svc.yaml
2026-03-14 09:55:30.507943 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/secrets/watcher/cert-watcher-public-svc.yaml
2026-03-14 09:55:30.507961 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/
2026-03-14 09:55:30.507977 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/alertmanager-operated.yaml
2026-03-14 09:55:30.507997 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/ceilometer-internal.yaml
2026-03-14 09:55:30.508029 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/keystone-internal.yaml
2026-03-14 09:55:30.508052 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/keystone-public.yaml
2026-03-14 09:55:30.508069 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/kube-state-metrics.yaml
2026-03-14 09:55:30.508085 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/memcached.yaml
2026-03-14 09:55:30.508100 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/metric-storage-alertmanager.yaml
2026-03-14 09:55:30.508116 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/metric-storage-prometheus.yaml
2026-03-14 09:55:30.508133 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/metric-storage-thanos-sidecar.yaml
2026-03-14 09:55:30.508149 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/openstack-galera.yaml
2026-03-14 09:55:30.508165 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/openstack.yaml
2026-03-14 09:55:30.508181 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/prometheus-operated.yaml
2026-03-14 09:55:30.508196 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/rabbitmq-nodes.yaml
2026-03-14 09:55:30.508649 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/rabbitmq-notifications-nodes.yaml
2026-03-14 09:55:30.508673 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/rabbitmq-notifications.yaml
2026-03-14 09:55:30.508694 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/services/rabbitmq.yaml
2026-03-14 09:55:30.508712 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/
2026-03-14 09:55:30.508728 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/alertmanager-metric-storage.yaml
2026-03-14 09:55:30.508744 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/ceilometer.yaml
2026-03-14 09:55:30.508760 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/kube-state-metrics.yaml
2026-03-14 09:55:30.508776 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/memcached.yaml
2026-03-14 09:55:30.508792 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/openstack-galera.yaml
2026-03-14 09:55:30.508807 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/prometheus-metric-storage.yaml
2026-03-14 09:55:30.508823 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/rabbitmq-notifications-server.yaml
2026-03-14 09:55:30.508838 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/statefulset/rabbitmq-server.yaml
2026-03-14 09:55:30.508853 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/namespaces/watcher-kuttl-default/subscriptions/
2026-03-14 09:55:30.508869 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/
2026-03-14 09:55:30.508884 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/l2advertisement.yaml
2026-03-14 09:55:30.508899 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/ipaddresspools/
2026-03-14 09:55:30.508915 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/ipaddresspools/ctlplane.yaml
2026-03-14 09:55:30.508934 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/ipaddresspools/designateext.yaml
2026-03-14 09:55:30.508950 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/ipaddresspools/internalapi.yaml
2026-03-14 09:55:30.509429 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/ipaddresspools/storage.yaml
2026-03-14 09:55:30.509624 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/ipaddresspools/tenant.yaml
2026-03-14 09:55:30.509643 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/nnce/
2026-03-14 09:55:30.509660 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/nnce/crc.ens7-crc.yaml
2026-03-14 09:55:30.509677 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/nncp/
2026-03-14 09:55:30.509694 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/nncp/ens7-crc.yaml
2026-03-14 09:55:30.509710 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/nns/
2026-03-14 09:55:30.509726 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/network/nns/crc.yaml
2026-03-14 09:55:30.509742 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/nodes/
2026-03-14 09:55:30.509758 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/nodes/crc.describe
2026-03-14 09:55:30.509774 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/nodes/crc.yaml
2026-03-14 09:55:30.509789 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/sos-reports/
2026-03-14 09:55:30.510140 | controller | cL+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/sos-reports/sos-report- -> ../_all_nodes/sosreport-
2026-03-14 09:55:30.510167 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/sos-reports/_all_nodes/
2026-03-14 09:55:30.510190 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/
2026-03-14 09:55:30.510207 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/
2026-03-14 09:55:30.510223 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/cert-manager-webhook.yaml
2026-03-14 09:55:30.510239 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/infra-operator-mutating-webhook-configuration.yaml
2026-03-14 09:55:30.510256 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/mwatcher-v1beta1.kb.io-7gpg8.yaml
2026-03-14 09:55:30.510883 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/mwatcherapi-v1beta1.kb.io-knbzs.yaml
2026-03-14 09:55:30.510911 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/mwatcherapplier-v1beta1.kb.io-kcnrn.yaml
2026-03-14 09:55:30.510929 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/mwatcherdecisionengine-v1beta1.kb.io-r9n8d.yaml
2026-03-14 09:55:30.510946 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/nmstate.yaml
2026-03-14 09:55:30.510963 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/openstack-baremetal-operator-mutating-webhook-configuration.yaml
2026-03-14 09:55:30.510990 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/mutating/openstack-operator-mutating-webhook-configuration.yaml
2026-03-14 09:55:30.511008 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/
2026-03-14 09:55:30.511060 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/alertmanagerconfigs.monitoring.rhobs-7f4nd.yaml
2026-03-14 09:55:30.511231 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/bfdprofilevalidationwebhook.metallb.io-n9ld7.yaml
2026-03-14 09:55:30.511255 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/bgpadvertisementvalidationwebhook.metallb.io-vjfr7.yaml
2026-03-14 09:55:30.511278 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/bgppeersvalidationwebhook.metallb.io-gdt9p.yaml
2026-03-14 09:55:30.511295 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/cert-manager-webhook.yaml
2026-03-14 09:55:30.511312 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/communityvalidationwebhook.metallb.io-4tr8v.yaml
2026-03-14 09:55:30.511329 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/controlplanemachineset.machine.openshift.io.yaml
2026-03-14 09:55:30.511345 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/frr-k8s-validating-webhook-configuration.yaml
2026-03-14 09:55:30.511361 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/infra-operator-validating-webhook-configuration.yaml
2026-03-14 09:55:30.511378 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/ipaddresspoolvalidationwebhook.metallb.io-8h6cz.yaml
2026-03-14 09:55:30.511394 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/l2advertisementvalidationwebhook.metallb.io-df5xd.yaml
2026-03-14 09:55:30.511549 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/metallbvalidationwebhook.metallb.io-sqdjd.yaml
2026-03-14 09:55:30.511569 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/monitoringconfigmaps.openshift.io.yaml
2026-03-14 09:55:30.511586 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/multus.openshift.io.yaml
2026-03-14 09:55:30.511602 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/network-node-identity.openshift.io.yaml
2026-03-14 09:55:30.511618 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/openstack-baremetal-operator-validating-webhook-configuration.yaml
2026-03-14 09:55:30.511634 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/openstack-operator-validating-webhook-configuration.yaml
2026-03-14 09:55:30.511658 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/prometheusrules.monitoring.rhobs-tm556.yaml
2026-03-14 09:55:30.511675 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/vwatcher-v1beta1.kb.io-ldlqb.yaml
2026-03-14 09:55:30.511691 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/vwatcherapi-v1beta1.kb.io-nfhx8.yaml
2026-03-14 09:55:30.511708 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/vwatcherapplier-v1beta1.kb.io-6jwk9.yaml
2026-03-14 09:55:30.511724 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7cdea36ddb29e0814006da6ef0a0dd0d2c44975a23ddabbe32663c6f88dc35f3/webhooks/validating/vwatcherdecisionengine-v1beta1.kb.io-dt9kv.yaml
2026-03-14 09:55:30.511908 | controller | cd+++++++++ registries.conf.d/
2026-03-14 09:55:30.511936 | controller | >f+++++++++ registries.conf.d/000-shortnames.conf
2026-03-14 09:55:30.511954 | controller | cd+++++++++ system-config/
2026-03-14 09:55:30.511972 | controller | cd+++++++++ system-config/libvirt/
2026-03-14 09:55:30.511987 | controller | >f+++++++++ system-config/libvirt/libvirt-admin.conf
2026-03-14 09:55:30.512003 | controller | >f+++++++++ system-config/libvirt/libvirt.conf
2026-03-14 09:55:30.856519 | controller | ok
2026-03-14 09:55:31.392200 | controller | ok
2026-03-14 09:55:31.479536 | 
2026-03-14 09:55:31.479665 | TASK [Return artifact to Zuul]
2026-03-14 09:55:31.506401 | controller | skipping: Conditional result was False
2026-03-14 09:55:31.665400 | 
2026-03-14 09:55:31.665511 | PLAY [Run ci/playbooks/collect-logs.yml on CRC host]
2026-03-14 09:55:31.687877 | 
2026-03-14 09:55:31.687990 | TASK [Get kubelet journalctl logs]
2026-03-14 09:55:32.731476 | crc | Output suppressed because no_log was given
2026-03-14 09:55:32.743108 | 
2026-03-14 09:55:32.743202 | TASK [Compress logs bigger than 2MB]
2026-03-14 09:55:33.044848 | crc | gzip: /var/home/core/zuul-output/logs/kubelet.log.gz already exists;	not overwritten
2026-03-14 09:55:33.305363 | crc | ERROR
2026-03-14 09:55:33.305796 | crc | {
2026-03-14 09:55:33.305975 | crc |   "delta": "0:00:00.007202",
2026-03-14 09:55:33.306280 | crc |   "end": "2026-03-14 09:55:33.045358",
2026-03-14 09:55:33.306360 | crc |   "msg": "non-zero return code",
2026-03-14 09:55:33.306433 | crc |   "rc": 1,
2026-03-14 09:55:33.306509 | crc |   "start": "2026-03-14 09:55:33.038156"
2026-03-14 09:55:33.306555 | crc | }
2026-03-14 09:55:33.306619 | crc | ERROR: Ignoring Errors
2026-03-14 09:55:33.325534 | 
2026-03-14 09:55:33.325694 | TASK [Copy files from workspace on node]
2026-03-14 09:55:33.384954 | crc | ok
2026-03-14 09:55:33.413848 | 
2026-03-14 09:55:33.413983 | TASK [fetch-output : Set log path for multiple nodes]
2026-03-14 09:55:33.458745 | crc | ok
2026-03-14 09:55:33.469564 | 
2026-03-14 09:55:33.469698 | TASK [fetch-output : Set log path for single node]
2026-03-14 09:55:33.495732 | crc | skipping: Conditional result was False
2026-03-14 09:55:33.505875 | 
2026-03-14 09:55:33.505956 | LOOP [fetch-output : Ensure local output dirs]
2026-03-14 09:55:33.790766 | crc -> localhost | ok: "/var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/logs/crc"
2026-03-14 09:55:33.791111 | crc -> localhost | changed: All items complete
2026-03-14 09:55:33.791147 | 
2026-03-14 09:55:34.023351 | crc -> localhost | ok: "/var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/artifacts"
2026-03-14 09:55:34.273127 | crc -> localhost | ok: "/var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/docs"
2026-03-14 09:55:34.295877 | 
2026-03-14 09:55:34.296065 | LOOP [fetch-output : Collect logs, artifacts and docs]
2026-03-14 09:55:35.029196 | crc | changed:
2026-03-14 09:55:35.029615 | crc | .d..t...... ./
2026-03-14 09:55:35.029657 | crc | >f+++++++++ kubelet.log
2026-03-14 09:55:35.555960 | crc | ok
2026-03-14 09:55:36.077875 | crc | ok
2026-03-14 09:55:36.136122 | 
2026-03-14 09:55:36.136293 | PLAY RECAP
2026-03-14 09:55:36.136361 | controller | ok: 15 changed: 11 unreachable: 0 failed: 0 skipped: 7 rescued: 0 ignored: 1
2026-03-14 09:55:36.136400 | crc | ok: 5 changed: 3 unreachable: 0 failed: 0 skipped: 1 rescued: 0 ignored: 1
2026-03-14 09:55:36.136426 | 
2026-03-14 09:55:36.347768 | POST-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml@main]
2026-03-14 09:55:36.353733 | POST-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-autohold.yml@main]
2026-03-14 09:55:36.920602 | 
2026-03-14 09:55:36.920710 | PLAY [Run ci/playbooks/multinode-autohold.yml]
2026-03-14 09:55:37.000541 | 
2026-03-14 09:55:37.000664 | TASK [Verify if "success" flag exists after successful tests execution]
2026-03-14 09:55:37.597292 | controller | ok
2026-03-14 09:55:37.621297 | 
2026-03-14 09:55:37.621404 | TASK [Fetch existing autoholds from zuul]
2026-03-14 09:55:37.680966 | controller | ok
2026-03-14 09:55:37.731649 | 
2026-03-14 09:55:37.731745 | TASK [krb_request : Perform the request]
2026-03-14 09:55:38.351212 | controller | ok
2026-03-14 09:55:38.439222 | 
2026-03-14 09:55:38.439359 | TASK [Check if any autohold matches]
2026-03-14 09:55:38.536397 | controller | skipping: Conditional result was False
2026-03-14 09:55:38.670810 | 
2026-03-14 09:55:38.670984 | PLAY RECAP
2026-03-14 09:55:38.671053 | controller | ok: 2 changed: 0 unreachable: 0 failed: 0 skipped: 1 rescued: 0 ignored: 0
2026-03-14 09:55:38.671079 | 
2026-03-14 09:55:38.843300 | POST-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-autohold.yml@main]
2026-03-14 09:55:38.853236 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/print-machineconfigpool.yaml@master]
2026-03-14 09:55:39.475456 | 
2026-03-14 09:55:39.475611 | PLAY [crc]
2026-03-14 09:55:39.503358 | 
2026-03-14 09:55:39.503478 | TASK [Print machineconfigpool]
2026-03-14 09:55:40.335081 | crc | NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
2026-03-14 09:55:40.335186 | crc | master   rendered-master-f83a149c7490c3b0bf850df839837383   False     False      True       1              0                   0                     1                      384d
2026-03-14 09:55:40.335203 | crc | worker   rendered-worker-f1c7445993afc50ca9b63f2194fcd569   True      False      False      0              0                   0                     0                      384d
2026-03-14 09:55:40.543928 | crc | changed
2026-03-14 09:55:40.572036 | 
2026-03-14 09:55:40.572157 | PLAY RECAP
2026-03-14 09:55:40.572209 | crc | ok: 1 changed: 1 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0
2026-03-14 09:55:40.572231 | 
2026-03-14 09:55:40.716834 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/print-machineconfigpool.yaml@master]
2026-03-14 09:55:40.722735 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/compress-crc-cloud-logs.yaml@master]
2026-03-14 09:55:41.314203 | 
2026-03-14 09:55:41.314320 | PLAY [crc]
2026-03-14 09:55:41.342220 | 
2026-03-14 09:55:41.342308 | TASK [Create compressed log tar gz file - crc-cloud logs]
2026-03-14 09:55:42.042711 | crc | tar: Removing leading `/' from member names
2026-03-14 09:55:42.046692 | crc | tar: /var/home/core/workdir/*log: Cannot stat: No such file or directory
2026-03-14 09:55:42.046741 | crc | tar: Exiting with failure status due to previous errors
2026-03-14 09:55:42.382530 | crc | ok
2026-03-14 09:55:42.400703 | 
2026-03-14 09:55:42.400925 | TASK [Copy crc-cloud logs]
2026-03-14 09:55:42.770925 | crc | cp: cannot stat '/var/home/core/workdir/*log': No such file or directory
2026-03-14 09:55:42.962493 | crc | ok
2026-03-14 09:55:42.996388 | 
2026-03-14 09:55:42.996494 | PLAY RECAP
2026-03-14 09:55:42.996542 | crc | ok: 2 changed: 0 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0
2026-03-14 09:55:42.996569 | 
2026-03-14 09:55:43.113888 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/compress-crc-cloud-logs.yaml@master]
2026-03-14 09:55:43.128238 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/compress-logs.yaml@master]
2026-03-14 09:55:43.856808 | 
2026-03-14 09:55:43.856948 | PLAY [all]
2026-03-14 09:55:43.949615 | 
2026-03-14 09:55:43.949792 | TASK [Create compressed log tar gz file - ci-framework-data]
2026-03-14 09:55:44.626040 | controller | tar: Removing leading `/' from member names
2026-03-14 09:55:44.760740 | crc | tar: Removing leading `/' from member names
2026-03-14 09:55:44.760828 | crc | tar: /var/home/core/ci-framework-data: Cannot stat: No such file or directory
2026-03-14 09:55:44.763109 | crc | tar: Exiting with failure status due to previous errors
2026-03-14 09:55:44.765388 | crc | mv: cannot move '/var/home/core/ci-framework-data-crc-all-logs.tar.gz' to '/var/home/core/ci-framework-data/logs/ci-framework-data-crc-all-logs.tar.gz': No such file or directory
2026-03-14 09:55:44.766980 | crc | chown: cannot access '/var/home/core/ci-framework-data/logs/ci-framework-data-crc-all-logs.tar.gz': No such file or directory
2026-03-14 09:55:45.132219 | crc | ERROR
2026-03-14 09:55:45.132551 | crc | {
2026-03-14 09:55:45.132619 | crc |   "delta": "0:00:00.012612",
2026-03-14 09:55:45.132665 | crc |   "end": "2026-03-14 09:55:44.767242",
2026-03-14 09:55:45.132704 | crc |   "msg": "non-zero return code",
2026-03-14 09:55:45.132742 | crc |   "rc": 1,
2026-03-14 09:55:45.132780 | crc |   "start": "2026-03-14 09:55:44.754630"
2026-03-14 09:55:45.132816 | crc | }
2026-03-14 09:55:45.132870 | crc | ERROR: Ignoring Errors
2026-03-14 09:55:53.523151 | controller | ok
2026-03-14 09:55:53.553991 | 
2026-03-14 09:55:53.554109 | TASK [Create compressed log tar gz file - zuul-output]
2026-03-14 09:55:53.983513 | controller | tar: Removing leading `/' from member names
2026-03-14 09:55:54.027516 | crc | tar: Removing leading `/' from member names
2026-03-14 09:55:54.265903 | crc | ok
2026-03-14 09:55:56.891108 | controller | ok
2026-03-14 09:55:57.056896 | 
2026-03-14 09:55:57.057088 | PLAY RECAP
2026-03-14 09:55:57.057140 | controller | ok: 2 changed: 0 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0
2026-03-14 09:55:57.057167 | crc | ok: 2 changed: 0 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 1
2026-03-14 09:55:57.057185 | 
2026-03-14 09:55:57.156862 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/compress-logs.yaml@master]
2026-03-14 09:55:57.161347 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/bootstraped-ci-network-cleanup.yml@master]
2026-03-14 09:55:57.737647 | 
2026-03-14 09:55:57.737796 | PLAY [Run playbooks/crc/bootstraped-ci-network-cleanup.yml]
2026-03-14 09:55:57.816138 | 
2026-03-14 09:55:57.816284 | TASK [Create openstack config dir]
2026-03-14 09:55:58.449702 | controller | ok
2026-03-14 09:55:58.474531 | 
2026-03-14 09:55:58.474601 | TASK [Check for cleanup skip flag file presence]
2026-03-14 09:55:58.896839 | controller | ok
2026-03-14 09:55:58.928669 | 
2026-03-14 09:55:58.928812 | TASK [Generate clouds config from cloud_secrets secret]
2026-03-14 09:55:59.791295 | controller | changed
2026-03-14 09:55:59.823797 | 
2026-03-14 09:55:59.823867 | LOOP [Delete ports from each host]
2026-03-14 09:55:59.936982 | controller | ok: "crc"
2026-03-14 09:56:02.924805 | controller | ok: "controller"
2026-03-14 09:56:02.989672 | controller | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstraped-ci-network-cleanup-instance-ports.yml
2026-03-14 09:56:02.994498 | controller | included: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstraped-ci-network-cleanup-instance-ports.yml
2026-03-14 09:56:03.028070 | 
2026-03-14 09:56:03.028163 | TASK [Fetch port info]
2026-03-14 09:56:05.211823 | controller | admin_state_up: true
2026-03-14 09:56:05.211976 | controller | allowed_address_pairs: []
2026-03-14 09:56:05.212146 | controller | binding_host_id: null
2026-03-14 09:56:05.212248 | controller | binding_profile: {}
2026-03-14 09:56:05.212406 | controller | binding_vif_details: {}
2026-03-14 09:56:05.212543 | controller | binding_vif_type: null
2026-03-14 09:56:05.212744 | controller | binding_vnic_type: normal
2026-03-14 09:56:05.212885 | controller | created_at: '2026-03-14T08:54:17Z'
2026-03-14 09:56:05.213023 | controller | data_plane_status: null
2026-03-14 09:56:05.213146 | controller | description: ''
2026-03-14 09:56:05.213305 | controller | device_id: 6518bcde-aa50-4603-92c7-71dcf31294f9
2026-03-14 09:56:05.213468 | controller | device_owner: compute:nova
2026-03-14 09:56:05.213597 | controller | device_profile: null
2026-03-14 09:56:05.213986 | controller | dns_assignment:
2026-03-14 09:56:05.214109 | controller | - fqdn: host-192-168-122-10.openstacklocal.
2026-03-14 09:56:05.214118 | controller |   hostname: host-192-168-122-10
2026-03-14 09:56:05.214123 | controller |   ip_address: 192.168.122.10
2026-03-14 09:56:05.214129 | controller | dns_domain: ''
2026-03-14 09:56:05.214232 | controller | dns_name: ''
2026-03-14 09:56:05.214359 | controller | extra_dhcp_opts: []
2026-03-14 09:56:05.214593 | controller | fixed_ips:
2026-03-14 09:56:05.214709 | controller | - ip_address: 192.168.122.10
2026-03-14 09:56:05.214718 | controller |   subnet_id: a281b98f-17e4-4b27-b77d-2827e0a63895
2026-03-14 09:56:05.214724 | controller | hardware_offload_type: null
2026-03-14 09:56:05.214792 | controller | hints: ''
2026-03-14 09:56:05.214903 | controller | id: 022709b1-dd59-437e-b00d-049cb0e9bf02
2026-03-14 09:56:05.215000 | controller | ip_allocation: immediate
2026-03-14 09:56:05.215102 | controller | mac_address: fa:16:3e:e3:c1:c5
2026-03-14 09:56:05.215225 | controller | name: crc-6518bcde-aa50-4603-92c7-71dcf31294f9
2026-03-14 09:56:05.215360 | controller | network_id: 9dc2d874-9779-4639-a8a6-76b84f2f8f52
2026-03-14 09:56:05.215459 | controller | numa_affinity_policy: null
2026-03-14 09:56:05.215587 | controller | port_security_enabled: false
2026-03-14 09:56:05.215718 | controller | project_id: 4b633c451ac74233be3721a3635275e5
2026-03-14 09:56:05.215818 | controller | propagate_uplink_status: null
2026-03-14 09:56:05.215913 | controller | resource_request: null
2026-03-14 09:56:05.216033 | controller | revision_number: 4
2026-03-14 09:56:05.216134 | controller | qos_network_policy_id: null
2026-03-14 09:56:05.216228 | controller | qos_policy_id: null
2026-03-14 09:56:05.216340 | controller | security_group_ids: []
2026-03-14 09:56:05.216430 | controller | status: ACTIVE
2026-03-14 09:56:05.216517 | controller | tags: []
2026-03-14 09:56:05.216659 | controller | trunk_details: null
2026-03-14 09:56:05.216780 | controller | trusted: null
2026-03-14 09:56:05.216900 | controller | updated_at: '2026-03-14T08:54:33Z'
2026-03-14 09:56:05.683599 | controller | changed
2026-03-14 09:56:05.714424 | 
2026-03-14 09:56:05.714539 | TASK [Set port info yaml for further usage]
2026-03-14 09:56:05.818408 | controller | ok
2026-03-14 09:56:05.877042 | 
2026-03-14 09:56:05.877178 | TASK [Detach port]
2026-03-14 09:56:08.504116 | controller | changed
2026-03-14 09:56:08.557713 | 
2026-03-14 09:56:08.557842 | LOOP [Delete port association with trunk network]
2026-03-14 09:56:08.668945 | 
2026-03-14 09:56:08.669177 | LOOP [Delete trunk sub ports (tagged ones)]
2026-03-14 09:56:08.785468 | 
2026-03-14 09:56:08.785680 | TASK [Delete the trunk network itself]
2026-03-14 09:56:09.456129 | controller | skipping: Conditional result was False
2026-03-14 09:56:09.511729 | 
2026-03-14 09:56:09.511867 | TASK [Remove the parent/trunk port itself]
2026-03-14 09:56:22.727557 | controller | changed
2026-03-14 09:56:22.758991 | 
2026-03-14 09:56:22.759190 | TASK [Fetch port info]
2026-03-14 09:56:31.961574 | controller | admin_state_up: true
2026-03-14 09:56:31.961726 | controller | allowed_address_pairs: []
2026-03-14 09:56:31.961927 | controller | binding_host_id: null
2026-03-14 09:56:31.962031 | controller | binding_profile: {}
2026-03-14 09:56:31.962135 | controller | binding_vif_details: {}
2026-03-14 09:56:31.962388 | controller | binding_vif_type: null
2026-03-14 09:56:31.962527 | controller | binding_vnic_type: normal
2026-03-14 09:56:31.962641 | controller | created_at: '2026-03-14T08:53:56Z'
2026-03-14 09:56:31.962742 | controller | data_plane_status: null
2026-03-14 09:56:31.962838 | controller | description: ''
2026-03-14 09:56:31.962951 | controller | device_id: a1239aa3-61dc-4384-a93b-9c354d165e80
2026-03-14 09:56:31.963055 | controller | device_owner: compute:nova
2026-03-14 09:56:31.963164 | controller | device_profile: null
2026-03-14 09:56:31.963465 | controller | dns_assignment:
2026-03-14 09:56:31.963553 | controller | - fqdn: host-192-168-122-11.openstacklocal.
2026-03-14 09:56:31.963561 | controller |   hostname: host-192-168-122-11
2026-03-14 09:56:31.963566 | controller |   ip_address: 192.168.122.11
2026-03-14 09:56:31.963573 | controller | dns_domain: ''
2026-03-14 09:56:31.963645 | controller | dns_name: ''
2026-03-14 09:56:31.963736 | controller | extra_dhcp_opts: []
2026-03-14 09:56:31.963970 | controller | fixed_ips:
2026-03-14 09:56:31.964067 | controller | - ip_address: 192.168.122.11
2026-03-14 09:56:31.964077 | controller |   subnet_id: a281b98f-17e4-4b27-b77d-2827e0a63895
2026-03-14 09:56:31.964086 | controller | hardware_offload_type: null
2026-03-14 09:56:31.964154 | controller | hints: ''
2026-03-14 09:56:31.964265 | controller | id: 3fb3d0b3-3be6-4ea1-b59d-3121f46e758b
2026-03-14 09:56:31.964380 | controller | ip_allocation: immediate
2026-03-14 09:56:31.964495 | controller | mac_address: fa:16:3e:ac:8d:09
2026-03-14 09:56:31.964640 | controller | name: controller-a1239aa3-61dc-4384-a93b-9c354d165e80
2026-03-14 09:56:31.964756 | controller | network_id: 9dc2d874-9779-4639-a8a6-76b84f2f8f52
2026-03-14 09:56:31.964856 | controller | numa_affinity_policy: null
2026-03-14 09:56:31.964989 | controller | port_security_enabled: false
2026-03-14 09:56:31.965106 | controller | project_id: 4b633c451ac74233be3721a3635275e5
2026-03-14 09:56:31.965204 | controller | propagate_uplink_status: null
2026-03-14 09:56:31.965313 | controller | resource_request: null
2026-03-14 09:56:31.965459 | controller | revision_number: 4
2026-03-14 09:56:31.965582 | controller | qos_network_policy_id: null
2026-03-14 09:56:31.965680 | controller | qos_policy_id: null
2026-03-14 09:56:31.965776 | controller | security_group_ids: []
2026-03-14 09:56:31.965865 | controller | status: ACTIVE
2026-03-14 09:56:31.965949 | controller | tags: []
2026-03-14 09:56:31.966045 | controller | trunk_details: null
2026-03-14 09:56:31.966135 | controller | trusted: null
2026-03-14 09:56:31.966410 | controller | updated_at: '2026-03-14T08:54:10Z'
2026-03-14 09:56:32.428501 | controller | changed
2026-03-14 09:56:32.480280 | 
2026-03-14 09:56:32.480428 | TASK [Set port info yaml for further usage]
2026-03-14 09:56:32.610518 | controller | ok
2026-03-14 09:56:32.633166 | 
2026-03-14 09:56:32.633255 | TASK [Detach port]
2026-03-14 09:56:45.805825 | controller | changed
2026-03-14 09:56:45.838254 | 
2026-03-14 09:56:45.838358 | LOOP [Delete port association with trunk network]
2026-03-14 09:56:45.938989 | 
2026-03-14 09:56:45.939229 | LOOP [Delete trunk sub ports (tagged ones)]
2026-03-14 09:56:46.029894 | 
2026-03-14 09:56:46.030151 | TASK [Delete the trunk network itself]
2026-03-14 09:56:46.691683 | controller | skipping: Conditional result was False
2026-03-14 09:56:46.765843 | 
2026-03-14 09:56:46.766003 | TASK [Remove the parent/trunk port itself]
2026-03-14 09:56:59.555006 | controller | changed
2026-03-14 09:56:59.587327 | 
2026-03-14 09:56:59.587413 | TASK [Dettach router from subnet]
2026-03-14 09:57:00.208703 | controller | skipping: Conditional result was False
2026-03-14 09:57:00.294155 | 
2026-03-14 09:57:00.294297 | TASK [Delete router]
2026-03-14 09:57:00.925005 | controller | skipping: Conditional result was False
2026-03-14 09:57:00.978657 | 
2026-03-14 09:57:00.978847 | TASK [Delete subnet]
2026-03-14 09:57:13.656516 | controller | changed
2026-03-14 09:57:13.708564 | 
2026-03-14 09:57:13.708671 | TASK [Delete network]
2026-03-14 09:57:28.389928 | controller | changed
2026-03-14 09:57:28.438888 | 
2026-03-14 09:57:28.438976 | TASK [Remove clouds.yml secret]
2026-03-14 09:57:28.740218 | controller | changed
2026-03-14 09:57:28.871629 | 
2026-03-14 09:57:28.871754 | PLAY RECAP
2026-03-14 09:57:28.871797 | controller | ok: 16 changed: 10 unreachable: 0 failed: 0 skipped: 8 rescued: 0 ignored: 0
2026-03-14 09:57:28.871818 | 
2026-03-14 09:57:28.982660 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/bootstraped-ci-network-cleanup.yml@master]
2026-03-14 09:57:28.992785 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/ci-framework-rdo-base/dlrn/dlrn-report.yaml@master]
2026-03-14 09:57:29.598450 | 
2026-03-14 09:57:29.598619 | PLAY [Report DLRN results after job run]
2026-03-14 09:57:29.676204 | 
2026-03-14 09:57:29.676333 | TASK [Report job status to DLRN]
2026-03-14 09:57:29.747036 | controller | ok
2026-03-14 09:57:29.806631 | 
2026-03-14 09:57:29.806742 | TASK [dlrn_report : Warning if DLRN is not running]
2026-03-14 09:57:29.881917 | controller | skipping: Conditional result was False
2026-03-14 09:57:29.922186 | 
2026-03-14 09:57:29.922362 | TASK [dlrn_report : Install dlrnapi-client shyaml package]
2026-03-14 09:57:29.990500 | controller | skipping: Conditional result was False
2026-03-14 09:57:30.041652 | 
2026-03-14 09:57:30.041820 | TASK [dlrn_report : Install kinit related package]
2026-03-14 09:57:30.118751 | controller | skipping: Conditional result was False
2026-03-14 09:57:30.171699 | 
2026-03-14 09:57:30.171865 | TASK [dlrn_report : Install dlrn kerberos related packages]
2026-03-14 09:57:30.248338 | controller | skipping: Conditional result was False
2026-03-14 09:57:30.332954 | 
2026-03-14 09:57:30.333123 | TASK [Set zuul-log-path fact]
2026-03-14 09:57:30.388698 | controller | skipping: Conditional result was False
2026-03-14 09:57:30.415507 | 
2026-03-14 09:57:30.415616 | TASK [dlrn_report : Set the the value of cifmw_repo_setup_promotion when multiple dlrn tags are used]
2026-03-14 09:57:30.471351 | controller | skipping: Conditional result was False
2026-03-14 09:57:30.499281 | 
2026-03-14 09:57:30.499398 | TASK [Get hash related data from repo_setup role]
2026-03-14 09:57:30.555954 | controller | skipping: Conditional result was False
2026-03-14 09:57:30.582310 | 
2026-03-14 09:57:30.582441 | TASK [dlrn_report : Perform kinit for DLRN kerberos authentication]
2026-03-14 09:57:31.251765 | controller | skipping: Conditional result was False
2026-03-14 09:57:31.286139 | 
2026-03-14 09:57:31.286288 | TASK [dlrn_report : Set empty value for dlrnapi password]
2026-03-14 09:57:31.362453 | controller | skipping: Conditional result was False
2026-03-14 09:57:31.401199 | 
2026-03-14 09:57:31.401377 | TASK [dlrn_report : Report results to dlrn for the tested hash]
2026-03-14 09:57:32.091554 | controller | skipping: Conditional result was False
2026-03-14 09:57:32.282506 | 
2026-03-14 09:57:32.282627 | PLAY RECAP
2026-03-14 09:57:32.282666 | controller | ok: 0 changed: 0 unreachable: 0 failed: 0 skipped: 10 rescued: 0 ignored: 0
2026-03-14 09:57:32.282686 | 
2026-03-14 09:57:32.399123 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/ci-framework-rdo-base/dlrn/dlrn-report.yaml@master]
2026-03-14 09:57:32.403526 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master]
2026-03-14 09:57:33.003536 | 
2026-03-14 09:57:33.003685 | PLAY [all]
2026-03-14 09:57:33.119585 | 
2026-03-14 09:57:33.119751 | TASK [include_role : fetch-output]
2026-03-14 09:57:33.192482 | controller | ok
2026-03-14 09:57:33.199323 | crc | ok
2026-03-14 09:57:33.258183 | 
2026-03-14 09:57:33.258288 | TASK [fetch-output : Set log path for multiple nodes]
2026-03-14 09:57:33.369556 | controller | ok
2026-03-14 09:57:33.378800 | crc | ok
2026-03-14 09:57:33.402381 | 
2026-03-14 09:57:33.402448 | TASK [fetch-output : Set log path for single node]
2026-03-14 09:57:33.439187 | controller | skipping: Conditional result was False
2026-03-14 09:57:33.449869 | crc | skipping: Conditional result was False
2026-03-14 09:57:33.514105 | 
2026-03-14 09:57:33.514242 | LOOP [fetch-output : Ensure local output dirs]
2026-03-14 09:57:33.881785 | controller -> localhost | ok: "/var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/logs/controller"
2026-03-14 09:57:33.904474 | crc -> localhost | ok: "/var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/logs/crc"
2026-03-14 09:57:34.083489 | controller -> localhost | ok: "/var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/artifacts"
2026-03-14 09:57:34.149646 | crc -> localhost | ok: "/var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/artifacts"
2026-03-14 09:57:34.313137 | controller -> localhost | ok: "/var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/docs"
2026-03-14 09:57:34.372704 | crc -> localhost | ok: "/var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/docs"
2026-03-14 09:57:34.372832 | crc -> localhost | ok: All items complete
2026-03-14 09:57:34.372885 | 
2026-03-14 09:57:34.428949 | 
2026-03-14 09:57:34.429114 | LOOP [fetch-output : Collect logs, artifacts and docs]
2026-03-14 09:57:35.069331 | crc | changed:
2026-03-14 09:57:35.069610 | crc | .d..t...... ./
2026-03-14 09:57:35.069650 | crc | >f+++++++++ crc-cloud-workdir-crc-all-logs.tar.gz
2026-03-14 09:57:35.069679 | crc | >f+++++++++ zuul-output-crc-all-logs.tar.gz
2026-03-14 09:57:35.069706 | crc | cd+++++++++ crc-cloud/
2026-03-14 09:57:35.538680 | crc | ok
2026-03-14 09:57:35.569627 | controller | changed:
2026-03-14 09:57:35.569693 | controller | .d..t...... ./
2026-03-14 09:57:35.569720 | controller | >f+++++++++ zuul-output-controller-all-logs.tar.gz
2026-03-14 09:57:35.991376 | controller | ok
2026-03-14 09:57:36.046267 | crc | ok
2026-03-14 09:57:36.529105 | controller | ok
2026-03-14 09:57:36.529313 | controller | changed: All items complete
2026-03-14 09:57:36.529353 | 
2026-03-14 09:57:36.641239 | 
2026-03-14 09:57:36.641376 | TASK [include_role : fetch-output-openshift]
2026-03-14 09:57:36.670703 | controller | skipping: Conditional result was False
2026-03-14 09:57:36.681590 | crc | skipping: Conditional result was False
2026-03-14 09:57:36.708288 | 
2026-03-14 09:57:36.708374 | LOOP [merge-output-to-logs : Move artifacts and docs to logs dir]
2026-03-14 09:57:37.183915 | controller -> localhost | ok: Item: artifacts Runtime: 0:00:00.013301
2026-03-14 09:57:37.492168 | controller -> localhost | ok: Item: docs Runtime: 0:00:00.014141
2026-03-14 09:57:37.749134 | 
2026-03-14 09:57:37.749316 | PLAY [all]
2026-03-14 09:57:37.829494 | 
2026-03-14 09:57:37.829588 | TASK [remove-build-sshkey : Remove the build SSH key from all nodes]
2026-03-14 09:57:38.501458 | controller | changed
2026-03-14 09:57:38.727456 | crc | changed
2026-03-14 09:57:38.940111 | 
2026-03-14 09:57:38.940242 | PLAY RECAP
2026-03-14 09:57:38.940287 | controller | ok: 5 changed: 3 unreachable: 0 failed: 0 skipped: 2 rescued: 0 ignored: 0
2026-03-14 09:57:38.940314 | crc | ok: 4 changed: 2 unreachable: 0 failed: 0 skipped: 2 rescued: 0 ignored: 0
2026-03-14 09:57:38.940404 | 
2026-03-14 09:57:39.057088 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master]
2026-03-14 09:57:39.066929 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-logs.yaml@master]
2026-03-14 09:57:39.799298 | 
2026-03-14 09:57:39.799437 | PLAY [localhost]
2026-03-14 09:57:39.817953 | 
2026-03-14 09:57:39.818056 | TASK [generate-zuul-manifest : Generate Zuul manifest]
2026-03-14 09:57:40.483517 | localhost | changed
2026-03-14 09:57:40.494462 | 
2026-03-14 09:57:40.494629 | TASK [generate-zuul-manifest : Return Zuul manifest URL to Zuul]
2026-03-14 09:57:40.533088 | localhost | ok
2026-03-14 09:57:40.552755 | 
2026-03-14 09:57:40.552910 | TASK [add-fileserver : Create SSH private key tempfile]
2026-03-14 09:57:40.990548 | localhost | changed
2026-03-14 09:57:41.001696 | 
2026-03-14 09:57:41.001856 | TASK [add-fileserver : Create SSH private key from secret]
2026-03-14 09:57:41.838822 | localhost | changed
2026-03-14 09:57:41.844309 | 
2026-03-14 09:57:41.844377 | TASK [add-fileserver : Add fileserver ssh key]
2026-03-14 09:57:42.313361 | localhost | Identity added: /var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/tmp/ansible.pbc5hq0y (/var/lib/zuul/builds/fbfdf80f16824a47b233a8a597241091/work/tmp/ansible.pbc5hq0y)
2026-03-14 09:57:42.313653 | localhost | ok: Runtime: 0:00:00.012601
2026-03-14 09:57:42.319755 | 
2026-03-14 09:57:42.319876 | TASK [add-fileserver : Remove SSH private key from disk]
2026-03-14 09:57:42.677405 | localhost | ok: Runtime: 0:00:00.010224
2026-03-14 09:57:42.718534 | 
2026-03-14 09:57:42.718690 | TASK [add-fileserver : Add fileserver to inventory]
2026-03-14 09:57:42.805910 | localhost | changed
2026-03-14 09:57:42.815134 | 
2026-03-14 09:57:42.815266 | TASK [add-fileserver : Add fileserver server to known hosts]
2026-03-14 09:57:43.228249 | localhost | changed
2026-03-14 09:57:43.266233 | 
2026-03-14 09:57:43.266308 | PLAY [localhost]
2026-03-14 09:57:43.283056 | 
2026-03-14 09:57:43.283142 | TASK [Generate bulk log download script]
2026-03-14 09:57:43.303129 | localhost | ok
2026-03-14 09:57:43.319410 | 
2026-03-14 09:57:43.319495 | TASK [local-log-download : Check API endpoint is defined]
2026-03-14 09:57:43.349341 | localhost | ok: All assertions passed
2026-03-14 09:57:43.355341 | 
2026-03-14 09:57:43.355421 | TASK [local-log-download : Create download script]
2026-03-14 09:57:43.837395 | localhost -> localhost | changed
2026-03-14 09:57:43.847873 | 
2026-03-14 09:57:43.847937 | TASK [Register quick-download link]
2026-03-14 09:57:43.866936 | localhost | ok
2026-03-14 09:57:43.882274 | 
2026-03-14 09:57:43.882331 | PLAY [logserver.rdoproject.org]
2026-03-14 09:57:43.892124 | 
2026-03-14 09:57:43.892183 | TASK [Set zuul-log-path fact]
2026-03-14 09:57:43.908123 | logserver.rdoproject.org | ok
2026-03-14 09:57:43.917584 | 
2026-03-14 09:57:43.917647 | TASK [set-zuul-log-path-fact : Set log path for a build]
2026-03-14 09:57:43.943759 | logserver.rdoproject.org | ok
2026-03-14 09:57:43.948868 | 
2026-03-14 09:57:43.948929 | TASK [upload-logs : Create log directories]
2026-03-14 09:57:44.840803 | logserver.rdoproject.org | changed
2026-03-14 09:57:44.845236 | 
2026-03-14 09:57:44.845324 | TASK [upload-logs : Ensure logs are readable before uploading]
2026-03-14 09:57:45.208734 | logserver.rdoproject.org -> localhost | ok: Runtime: 0:00:00.055530
2026-03-14 09:57:45.218865 | 
2026-03-14 09:57:45.219002 | TASK [upload-logs : Upload logs to log server]
2026-03-14 09:57:50.840591 | logserver.rdoproject.org | Output suppressed because no_log was given
2026-03-14 09:57:50.847199 | 
2026-03-14 09:57:50.847338 | LOOP [upload-logs : Compress console log and json output]
2026-03-14 09:57:50.920165 | logserver.rdoproject.org | skipping: Conditional result was False
2026-03-14 09:57:50.931070 | logserver.rdoproject.org | skipping: Conditional result was False
2026-03-14 09:57:50.943728 | 
2026-03-14 09:57:50.943907 | LOOP [upload-logs : Upload compressed console log and json output]
2026-03-14 09:57:50.993974 | logserver.rdoproject.org | skipping: Conditional result was False
2026-03-14 09:57:50.994465 | 
2026-03-14 09:57:50.997498 | logserver.rdoproject.org | skipping: Conditional result was False
2026-03-14 09:57:51.008643 | 
2026-03-14 09:57:51.008827 | LOOP [upload-logs : Upload console log and json output]